Using Google Mediapipe’s Pose Landmark Model, Pinocchio takes in a live video feed and allows the user to control the movements of the Bantam Tools NextDraw in order to draw with the movements of their right hand.
In Pinocchio, although the user controls the plotter through their movement, externally, it might seem a little bit like the user is being controlled by the ‘machine’ itself (being forced to dance, almost). We wanted to explore the idea of what it really means (or can mean) to “draw by hand,” and how potential integrations of technology with traditional art methods can transform our fundamental understandings of art creation altogether.
Although we weren’t able to expand on it in time for our demo due to limitations, I am currently developing a more advanced version of Pinocchio that utilizes more pose landmarks, such as using the distance between right elbow and right hip (landmarks 13 and 23) to determine line thickness, or requiring the user to click their heels (which we would determine from the distance between landmarks 29 and 30) in order for the plotter to raise the pen up and down. This complex version of Pinocchio will act as a puzzle of sorts for the user to figure out how exactly their movements and actions influence the ways in which the plotter draws.