Three pillars of the experience

DEFINING THE THREE PILLARS

Last week I reset the direction of the project from noise to sound. This week I had to answer the question that naturally comes next. Saying that the project visualises the sound of a place is not a plan. It is only a sentence. For this to become something I can actually build, I needed to break the experience down into concrete parts and decide what each part is responsible for.

By the end of the week the experience had three parts. I call them the three pillars.

① Audio Visualization. The sound data I collect from each district in Singapore has to be turned into something you can see. Particles, fluids, colour, motion. The goal is not to decorate the audio with visuals. The goal is to make the properties of the sound such as frequency, amplitude, and rhythm directly drive the form of what is on screen. When the audio changes, the image changes. When the audio stops, the image settles. If someone walks past the screen with no headphones, they should still be able to guess something about the place from the shape of the visuals alone.

② Virtual Tourism. The visitor is not looking at Singapore from outside. They are walking into it. Each district becomes an interactive 3D environment they can move through, not a 360 photograph they stand in front of. This is the part of the project that answers my target audience, the people who want to visit Singapore but cannot, and the people who want to learn about the city before they go. A flat image will not do that work. A space they can enter will.

③ Embodiment Interaction. The third pillar is the body. Mouse and keyboard are not enough. The visitor should be able to reach into the visualisation with their hands, shift their body to change what they hear, and feel like the space is responding to them the way a real place does. This is the pillar that turns the piece from a screen experience into something closer to a physical one.

Once the three pillars were named I spent time making sure I understood how they connect. The answer is at the convergence point at the centre. What happens when all three are active at the same time? A visitor is walking through the Bugis district. The market noise from the recording is driving thousands of particles around them. They reach out with their hand and the particles move out of the way. The sound shifts because they turned their head. All three pillars are firing at once and the visitor does not see them as three separate things. They see one experience. That is the target.

TECHNICAL FEASIBILITY RESEARCH

Defining the three pillars was the easy half. The harder question was whether I could actually build them. I spent the second half of the week doing technical research on each pillar, not to commit to a specific tool yet but to check that the pillars were not impossible.

For Audio Visualization I looked at particle systems, fluid simulations, and audio analysis methods like FFT for extracting frequency bands in real time. WebGL was the first option because I already had some JavaScript experience, but I also read about native approaches where a desktop application would have more direct access to the GPU. At this point I did not decide. I only confirmed that both routes exist and both can handle audio-reactive particles at a scale that might be interesting.

For Virtual Tourism the question was how to build a 3D space that the visitor can move through. Three.js came up immediately for the web, and I also looked at game engines and custom renderers for a native application. The bigger concern was not the rendering but the content. Each district needs a 3D model that feels like the place without being a tourist render. I did not start modelling anything this week, but I checked whether tools like Blender could produce the simplified style I had in mind.

For Embodiment Interaction I looked into MediaPipe. It tracks the hand in real time using only a webcam, no special hardware. For a graduation project this is important. I do not want the piece to require a headset or a controller to work. If the visitor sits down at a computer with a webcam, it should run. Other options exist but MediaPipe was the closest to zero-friction, and it has Python and JavaScript bindings so it fits either a web or native direction.

By the end of the research I had not chosen a single tool for anything. I had confirmed that every pillar has at least one realistic way to be built, and that there is a path where all three could run together in the same application. That was enough to close the week. The plan holds. I can build this. The next week would be the first attempt to actually put a version of the three pillars into a working prototype.