Mobile devices already come with a wide variety of hardware sensors, from simple cameras/microphones, accelerometers, and gyroscopes to more specialized hardware such as contact pressure sensors or relative humidity detectors. Although a typical device does not contain all of these potential sensors, there are several sensors that are reliably present even on older devices, simply due to basic system requirements. These include an accelerometer (which is typically used for automatic landscape/portrait screen rotation) and, for smartphones, a microphone and proximity sensor (to disable the touchscreen when the phone is held to the user’s ear). While not essential for core device functionality, virtually all modern smartphones and tablets also have a camera, although access to this sensor through PhoneIoT is handled differently due to privacy concerns (see
Section 3.1). Additionally, through services such as Google’s Fused Location Provider API, any mobile device connected to the Internet can retrieve live location data, if not by GPS, then by estimation from the connected local network.
The PhoneIoT app is capable of accessing all of these common sensors and more. If a sensor is not present on the device, is disabled, or otherwise blocked by app permissions, it is simply logically disabled as a target for IoT interactions through the NetsBlox interface. PhoneIoT continuously monitors data from all available sensors and makes it available to the NetsBlox server when requested by an authenticated student’s program. The server also handles other specialized requests, such as GUI configuration and forwarding user interactions as messages to linked NetsBlox clients.
Figure 1 visualizes this system architecture.
3.1. Privacy
Due to its extensive access to live device sensors, PhoneIoT introduces a number of potential privacy issues, primarily due to exposing access to live location data, the camera, and the microphone over the Internet. These concerns become especially important given that minors will be using the app, as it is largely meant to be a K-12 educational tool. For instance, it would not be acceptable for someone to be unknowingly tracked or spied upon through the NetsBlox interface due to forgetting to close the app. Because of this, unless explicitly requested with the “run in background” setting in the menu, the app ceases communication with the server and rejects all incoming requests upon being put into the background (minimized) or when the device’s display turns off, e.g., due to inactivity. As a further precaution, the app generates a random password which must be provided for any IoT interaction. For added security, this password is set to expire one day after generation (or upon user request), at which point a new random password is created, effectively cutting off any active connections. This one day window is sufficient for most uses of the app while still providing necessary privacy guarantees.
The password and expiry behavior is sufficient to make location data reasonably secure, but the camera and microphone could still be problematic. To solve the microphone issue, PhoneIoT only exposes the current volume level, rather than the actual waveform/content. To solve the camera issue, only images stored in image displays (described in
Section 3.3) are accessible; that is, the app does not allow a network request to take a new picture from the camera without user interaction. We believe these behaviors are sufficient to allow any K-12 audience to use the app while still affording them reasonable internet privacy.
3.2. Network Exchanges
As the app was meant to be used by young audiences in classroom settings, connecting PhoneIoT to a NetsBlox server deployment is very easy, only requiring a single button press from the app menu, which is already open when the app is started. Once pressed, the app connects to the server, announces its presence as an IoT device, provides the server with a unique identifier for further communications, and begins accepting network requests forwarded from the server.
The targeted server address is displayed as a URL in the app menu; this defaults to the primary NetsBlox server deployment but can be configured to any address. This is especially important in some classrooms around the world where a stable high-speed internet connection is not available, in which case a local deployment of NetsBlox can be used on the local network. The server address field is persistent across app restarts, so classrooms in these less-ideal circumstances would only have to configure their app settings once.
The UDP protocol was selected for PhoneIoT’s server interactions both for speed and because PhoneIoT’s data exchange model is already packet-based, making UDP a more natural model than streaming protocols such as TCP. Although UDP has the potential issue of dropping packets, for our purposes, this is actually desirable due to providing real-world lessons on error-handling in fallible network transactions. For instance, an early project for students could be to make robust wrappers for some PhoneIoT functions by repeating the operation until it succeeds.
The networking primitives used by the NetsBlox side of PhoneIoT are composed of “messages”, the same concept used throughout NetsBlox. In essence, a “message” is a structured block of data that is identified by name and has a set of fields associated with it. Messages can be sent with the “send msg” block and received (typically on a different computer) with a “when I receive” block. As an example, there is a default message type called “message” which has a single field called “msg”.
Figure 2 shows a simple example of how to send and receive a message of this type.
PhoneIoT provides two primary ways of accessing sensor data: polling for instantaneous values through explicit RPC requests, or streaming up to date values by registering a message type as a sensor update event that the device will send periodically based on the requested update interval. The explicit request style is similar to other pre-existing networking APIs in NetsBlox, and thus is a good introductory point for using PhoneIoT. However, in practice, many real-world IoT devices are accessed by continuous data streams, so lesson plans involving PhoneIoT quickly transition to this method.
Because PhoneIoT has the chance of dropped packets due to using the UDP protocol, performing polling in a loop can result in an error every once in a great while, depending on the network connection; this would have to be checked by students to avoid bugs in their code. Thus, streaming access can be introduced naturally as a more elegant solution, removing the need for both the loop and error checking code, as well as halving latency by inverting the problem and instructing the PhoneIoT device to send periodic update messages to the student’s project, rather than the project having to request each one. In this way, dropped packets go from causing errors to being simply absent update messages which are simply ignored by the student’s project without issue.
Figure 3 shows example code which registers for and receives sensor updates from the accelerometer every 100 ms.
From examining
Figure 3, one detail that may not be clear is how to know the name of the sensor to listen to, in this case “accelerometer”. For the most part, the names of the various sensors are identical to their polling RPCs; for instance, the “getLinearAcceleration” RPC has a matching sensor name of “linearAcceleration”. However, this is not always the case, in particular for soft sensors that PhoneIoT adds on top of the existing hardware sensors; for example, the “getCompassHeading” RPC’s sensor is in fact “orientation” (matching the “getOrientation” RPC). This information is available to users through the RPC help menu, which can be accessed by right clicking on a “call” block configured to, e.g., a polling RPC and selecting “help”.
Figure 4 shows the help information for the “getCompassHeading” (polling) RPC, which displays the name of the sensor/message type for the streaming access method, as well as the available fields that can be received on each update.
3.3. Custom GUI Controls
A key feature of PhoneIoT is its customizable interactive display. The static GUI for the main screen of the PhoneIoT app is intentionally minimalistic, containing only a button to toggle the pull-out app menu that contains all other static controls. Importantly, the menu is where the device’s id and password are shown, as well as the controls for connecting to the NetsBlox server; this menu is shown in
Figure 5a. When the menu is closed, the entirety of the screen, aside from the menu toggle button, is a single blank canvas which can be populated with content via various RPCs from the user’s program.
PhoneIoT supports many standard GUI control types, such as labels, buttons, text fields, image displays, and toggle switches, as well as some controls tailored for designing game controllers, such as virtual joysticks and touchpads. Each of these controls has fully customizable text content, location, size, color, orientation, and several other options depending on the specific control. A non-exhaustive example of custom control types is given in
Figure 5b,c.
An important consideration when designing PhoneIoT was to ensure that projects could be easily shared between students, who are potentially using different devices. Unfortunately, all of these different devices have different screen resolutions. To counteract this, PhoneIoT uses a relative, percent-based scale for specifying the x/y position and the width/height of controls. Specifically, PhoneIoT uses the standard GUI coordinate layout where is the top left corner of the canvas and is the bottom right. Additionally, the app automatically scales fonts depending on the DPI of the display, which allows fonts to be approximately the same size on all screens. These simple accommodations result in a coordinate system that is easy for students to use and is roughly invariant of the specific device display being used (up to mostly-minor aspect ratio stretching).
Being able to display custom content on the device is all well and good, but many of these types of controls are intended to facilitate receiving input from the user, such as button presses and joystick movement. To facilitate this, the various RPCs that are used to add controls to the device accept an optional configuration setting called “event”, which is the name of a message type that the device will send when a user interacts with the control. For instance, button events are triggered when pressed, text fields are triggered when the text is modified and submitted (i.e., there is not a separate update for each keystroke), image displays are triggered when a new image is saved in them from the camera (which can only be done manually by the user—see
Section 3.1), and joysticks/touchpads send an event when initially touched, continuously while moving (currently throttled to 10 Hz), and when released.
Figure 6 gives an example of creating a joystick control with an event called “joyMoved” that is triggered each time the stick is moved and displays the
x/
y position of the stick on the NetsBlox stage.
Message names for GUI events are customizable, but each type of control sends a different pre-determined set of values that users can receive by including them as fields on the message type. For instance, text field events include the new text, joystick/touchpad events include the
x/
y position, as well as a “tag” field that is one of “down”, “move”, or “up” to determine the type of interaction, and so on. Additionally, all events send the ID of the control (which can also be obtained by the return value of the RPC used to create the control) and the device ID on which the control was located (to differentiate controls on projects that configure multiple phones with the same GUI layout, such as a quiz game or group chat app). The specific information concerning what fields are available for each type of control can be found in the detailed documentation that is linked at the bottom of the (basic) RPC help menu. An example of this full, detailed documentation for the “addJoystick” RPC is shown in
Figure A1.
When a control is created, it is automatically assigned a control ID, which is returned by the RPC that was used to add the control (but if not needed, can be discarded by using a “run” block rather than a rounded “call” block). This control ID can then be used after creation to get and set state information about the specific control, or to delete the control while leaving all others intact. For instance, RPCs that contain text (e.g., labels, text fields, and buttons) can be used with the “getText” and “setText” RPCs, image displays can be used with “getImage” and “setImage”, toggle-based controls can be used with the “getToggleState” and “setToggleState” RPCs. This type of dynamic update and query behavior after control creation is vital, and can be used to perform tasks such as real-time updating of information displayed on the device screen.
This interactive component is important for teaching IoT to younger K-12 audiences because it immediately gives the students a useful tool related to things they already know, such as game controllers or content sharing with text/image displays. Due to how important phones are to today’s youth, introducing them to new ways of engaging with and controlling their devices can be especially motivating for continued interest in CS topics. The networking and IoT components are added to this to provide even more functionality and to teach the concepts to an already eager audience as a “side effect”.