David Glavas

Not so Social GAN

2020-10-02T12:44:00+00:00

TL;DR

We take a look at Social GAN’s pooling module, which according to the authors, helps the model “predict socially acceptable trajectories which avoid collisions”. Surprisingly, our experiments show that models without the pooling module tend to have just as much or less collisions than the models with the pooling module. We also show that models with a Random-Pooling function match the performance of the original Social GAN models with a Max-Pooling function. We conclude that Social GAN’s pooling module may not be as effective at encouraging socially acceptable trajectories as claimed in the paper.

While moving, people obey a large number of unwritten common sense rules that comply with social conventions. The ability to model these rules and use them to understand and predict human motion in complex real world environments is valuable for the development of socially aware systems. For example, an autonomous vehicle should be able to predict the future positions of pedestrians and adjust its path to avoid collisions. The problem of trajectory prediction can be viewed as a sequence generation task, where we are interested in predicting the future trajectories of people based on their past positions. In this case, future and past positions are sequences of coordinates (x, y tuples). Following the recent successes of Recurrent Neural Networks (RNNs) for sequence prediction tasks, the authors use an RNN Encoder-decoder framework to predict people’s future trajectories, given their past trajectories.

The main contributions of Social GAN are its approach to consider interactions between people (pooling module) and the way it encourages diverse sample generation (variety loss function). Next, we will take a closer look at Social GAN’s pooling module.

According to the authors, the pooling module causes the models to “predict socially acceptable trajectories which avoid collisions.” Surprisingly, the experiments that we will take a look at next contradict this claim. We will see that models without the pooling module tend to have just as much or less collisions than models with the pooling module.

The authors provide pre-trained models with and without the pooling module (the 20V-20 and 20VP-20 models from the paper). They trained two models—prediction lengths 8 and 12—for each of the five datasets. That’s twenty models in total, ten with and another ten without the pooling module. You can find these exact models here. Note that these are pre-trained by the author and we didn’t modify these models in any way. By using the evaluation script provided by the authors we get the same results they do.

Here we have one example situation where two pedestrians walk next to each other, the prediction differs from the ground truth but it looks socially acceptable. Note that colors differentiate pedestrians, dots are ground truth trajectories, and dashed lines are predictions of a model:

Each situation consists of a sequence of frames, each frame consists of the current position (x, y coordinates) of all pedestrians in the situation. We say that the situation contains a collision if there is a collision in any of the frames. We say that there is a collision in a frame if there are at least two pedestrians with a distance lower than a given threshold. So for a higher threshold we expect more collisions, for a high enough threshold we expect all situations to contain collisions. We are interested in a threshold for which we can say that resulting collisions are not socially acceptable. Then we can use this threshold to detect predictions that are not socially acceptable.

For each frame we compute the euclidian distance between all pedestrians, if any of these distances is below the threshold we classify the situation that this frame belongs to as containing a collision. The following figure shows the percentage of situations that contain collisions for different thresholds, for all of the pretrained models that the authors published (20V-20 and 20VP-20 with prediction lengths 8 and 12). Each of the four charts describes five of the in total twenty models that the authors provided:

Interestingly, the pooling module doesn’t seem to reduce the number of collisions. The models without the pooling module tend to have just as much or less collisions than the models with the pooling module. This contradicts the rationale behind the pooling module which is supposed to avoid collisions. As expected, we see that the more crowded datasets UNIV and ZARA2 have more collisions. This makes sense since there is less space when there are more pedestrians passing each other, so we expect them to get closer to each other. Also as expected, we see that the amount of collisions doesn’t change significantly between the prediction lengths, in fact, we see that a larger prediction length tends to result in more collisions which makes sense since for longer predicted trajectories there are more chances for collisions.

Next we are interested in inspecting specific situations which have been classified as having a collision. We evaluate the pretrained 20VP-20 model that is provided by the authors on ETH (prediction length 8). Let’s take a look at two arbitrarily chosen situations that were classified as having a collision (threshold = 0.1):

We see that despite the pooling module, the predicted trajectories deviate from the ground truth in a way that’s not socially acceptable. In the first example, the trajectories of the two pedestrians converge into one trajectory. In the second example, the blue and green pedestrians collide while crossing each other’s path.

As described in the paper, Social GAN uses Max-Pooling as the pooling function. To see how important a specific pooling function is, we replace the Max-Pooling with Mean-Pooling (instead of selecting the maximum value simply compute the mean over all values) and Random-Pooling (instead of selecting the maximum value simply randomly select one of the values). Note that we only swap out the max pooling function, we leave the rest of the pooling module as well as the training and evaluation procedures as is. For the different pooling modules we compute the same two error metrics as in the paper: Average Displacement Error (ADE): average L2 distance between ground truth and the prediction over all predicted time steps, and Final Displacement Error (FDE): distance between the predicted final destination and the true final destination at the last time step. We add a third metric to evaluate the social acceptability of the predicted trajectories: the percentage of situations that contain a collision for a given threshold. The following table contains the 3 metrics across all datasets for the pretrained models provided by the authors (20V-20 and 20VP-20 with prediction lengths 8), the retrained 20VP-20 model (no modifications, simply retrained models from the paper with exactly the same arguments as used by the authors), and the 2 pooling module variations (exactly the same as original 20VP-20 models except for the swapped pooling function).

We see that none of the pooling functions improves the collision metric. Surprisingly, as measured by the ADE and FDE, the 20VP-20 model with the random pooling function matches the performance of the other models.

The code for the discussed experiments can be found here.

Create your first adversarial examples

2020-07-06T14:15:00+00:00

TL;DR

The goal of this post is to help you quickly create (craft, generate, construct, call it whatever you want) your first adversarial examples. We use Keras running on top of TensorFlow to train the target neural network, then we craft the adversarial examples and demonstrate their effect on the target network. Train the target network yourself by running this or download it here. Craft the adversarial examples by running this, make sure to have the target network in the same directory.

What are adversarial examples?

An adversarial example is an input to a machine learning (ML) model that has been intentionally designed to cause the model to malfunction. We will see how to generate such inputs for deep neural networks using one of the earliest methods, the fast gradient sign method (FGSM). But first, let’s motivate the study of adversarial examples by taking a look at some of the threats they pose to deployed models in the real world. Feel free to skip this part if you want to craft asap.

Most early adversarial example research was performed under unrealistic conditions, but recently there has been an increasing number of more practical attacks [1]. Many works assume the adversary to have full access to the target model (white-box), but Papernot et al. have shown that attacks are possible even if the adversary has no access to the underlying model (black-box) [2]. Even state of the art machine learning models that are being offered as a service have been shown to be vulnerable to such black-box attacks [3]. Most works assume a threat model in which the adversary can feed data directly into the classifier on a digital level, but researchers have shown that adversarial examples that are printed onto paper and are perceived through a camera by the target network, are also classified incorrectly [4]. Researchers even printed 3D adversarial objects that are robust towards viewpoint shifts, camera noise, and other natural transformations [5].

Sharif et al. create physical adversarial examples to deceive state of the art neural network based face detection and commercial face recognition systems [6]. These systems are widely used for various sensitive purposes such as surveillance and access control. They print a pair of eyeglass frames, which allows the adversary that wears them to evade being recognized or to impersonate another individual. Their attack is physically realizable and inconspicuous, meaning that they create not a digital, but a physical adversarial accessory which doesn’t attract the attention of humans (eg. security guard), but which effectively turns the carrier of the accessory into an adversarial example.

Another interesting practical attack involves the use of adversarial examples to deceive road sign recognition network [7]. Eykholt et al. apply stickers to road signs which cause the target network to interpret a physical stop sign as a speed limit 45 sign. They show that attackers can physically modify objects such as road signs to reliably to cause classification errors in deep learning based systems under widely varying distances, angles, and resolutions.

Despite these threats, and despite the many approaches to protect neural networks that have been proposed, there is no known reliable defense against adversarial examples so far.

Train the target network

Adversarial examples need a target, some model to deceive. You can use any neural network really, but you might need to adapt the crafting process if the model’s interface changes. Here we train a Keras sequential model that achieves >99% accuracy on MNIST. You can either run the code and train the model yourself, or download the trained model and proceed to the next section. Note that the training might take well over an hour if you run TensorFlow on a CPU.

Craft the adversarial examples

At this point you should have a trained Keras sequential model stored on disk. You can craft the adversarial examples by running this (put the target model in the same directory as the script). The script loads the MNIST dataset, loads the trained model, crafts adversarial examples for the trained model from the MNIST test set, and stores the adversarial examples on disk. Congratulations, you have crafted 10000 adversarial examples, 61.31% of which cause the target model to return the wrong result. Note that this is the same model that achieves >99% accuracy on the original test set.

For a given example, FGSM computes the derivative of the model’s loss function with respect to each pixel, then it modifies each pixel in the direction of the gradient by a chosen perturbation size $\epsilon$. Given an example $x$, this method computes an adversarial example $x^*$ as

\[x^{*} = x + \epsilon sign(\nabla_x J(\Theta, x, y)),\]

where $J(\theta, x, y)$ is the target model’s loss function, with $\theta$ as the model’s parameters, and $y$ as the label of the given example $x$.

Let’s take a closer look at the effect of an adversarial example on the target network. On the left we see a natural, on the right an adversarial example:

Here is the output of the model’s softmax layer for the left (natural) example:

[[0.000000 0.000000 0.000000 0.000000 0.999992 0.000000 0.000000 0.000000 0.000000 0.000008]]

Here is the output for the right (adversarial) example:

[[0.051990 0.006389 0.050749 0.010629 0.253468 0.015621 0.035309 0.007679 0.120034 0.448132]]

We can interpret each of the numbers as the model’s certainty. The number at the i-th index is how certain the model is that the given example belongs to the i-th class. The index of the greatest number in this array corresponds to the predicted class. We can see that the network is fairly certain about the left image being a four, whereas for the right image it’s convinced that it’s a nine.

Note that generated adversarial examples don’t neccesarily succeed at deceiving the target network. Out of the 10000 adversarial examples we created, 38.69% fail at deceiving the target network. We can see one such case by examining the following example:

Here is the output for the left (natural) example:

[[0.000000 0.000000 1.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000]]

Here is the output for the right (adversarial) example:

[[0.163652 0.104641 0.446153 0.014896 0.021194 0.021860 0.091441 0.009633 0.096387 0.030143]]

We see that for the natural example, the model is certain that it’s a two. For the adversarial example, the model’s certainty that it’s a two drops significantly, but it still thinks it’s a two.

To conclude, we trained a neural network on MNIST that achieves >99% accuracy on the test set. Then we used FGSM to craft an adversarial test set for which the same network achieves 38.69% accuracy. In the end, we examined the effect of adversarial examples on the model’s softmax output. Note that there are more powerful attack algorithms that result in less perceptible perturbations of the image, and that cause the target model to make mistakes with much higher certainty. For example, performing the same experiment using Carlini and Wagner’s attack (C&W) results in the target model having less than 1% accuracy on the adversarial test set.

References:

[1]: Sun, Lu, Mingtian Tan, and Zhe Zhou. “A survey of practical adversarial example attacks.” Cybersecurity 1.1 (2018): 9.

[2]: Papernot, Nicolas, Patrick McDaniel, and Ian Goodfellow. “Transferability in machine learning: from phenomena to black-box attacks using adversarial samples.” arXiv preprint arXiv:1605.07277 (2016).

[3]: Papernot, Nicolas, et al. “Practical black-box attacks against machine learning.” Proceedings of the 2017 ACM on Asia conference on computer and communications security. ACM, 2017.

[4]: Kurakin, Alexey, Ian Goodfellow, and Samy Bengio. “Adversarial examples in the physical world.” arXiv preprint arXiv:1607.02533 (2016).

[5]: Athalye, Anish, et al. “Synthesizing robust adversarial examples.” arXiv preprint arXiv:1707.07397 (2017).

[6]: Sharif, Mahmood, et al. “Accessorize to a crime: Real and stealthy attacks on state-of-the-art face recognition.” Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security. ACM, 2016.

[7]: Eykholt, Kevin, et al. “Robust physical-world attacks on deep learning models.” arXiv preprint arXiv:1707.08945 (2017).

On Writing (Code) Well

2019-10-01T15:22:00+00:00

TL;DR

I point out similarities between programming and writing nonfiction based on interchangeable advice in Steve McConnell’s Code Complete and William Zinsser’s On Writing Well. For the identified similarities—clarity of thought, simplicity, and the importance of iterations—I elaborate on McConnell’s advice for writing code well.

The goal of this post is to share low-hanging fruits, that is, practical and immediately applicable advice any programmer can benefit from. I read On Writing Well and Code Complete in parallel which taught me some similarities. To ~~justify the time I spent procrastinating~~ keep this post interesting, I relate software construction to nonfiction writing and use the relationship as a basis for McConnell’s advice. First I’ll briefly introduce the books and give you a reason to believe the relationship exists. Then, I’ll go over the three main similarities—clarity of thought, simplicity, and the importance of iterations—that I found to be especially relevant for constructing software.

Let’s take a brief look at the two books.

The Books

Zinsser’s On Writing Well is all about expressing oneself with clarity, simplicity, brevity and humanity. It gives a glimpse into the habits of a professional writer and covers general advice such as perseverance, consistency, how to write a good leads and endings, how to not sound emotionless or like a copycat, and much more. He then shows how to apply this advice to various forms such as interviews, travel articles, memoirs, science, technology, business writing, humor and more.

McConnell’s Code Complete is a guided tour on lots of widely used development practices. It covers all kinds of issues related to software construction—from variables and statements to code tuning and collaborative construction. Besides learning a ton of new things, I enjoyed seeing tricks that I was using for some time now—especially those which I did unconsciously and never bothered to stop and think more about. To all those little things—like minimizing the distance in lines between the initialization of variables and their references—McConnell gives names such as live time and span. He manages to give names to things that most of us do intuitively but don’t consciously think about. Regardless of the specific names, the descriptions allow the reader to put a finger on what he already knows while picking up lots of new tricks along the way.

So is there a connection between programming and nonfiction writing? If so, then we should be able to find parts in the books with the same underlying ideas.

For the following four quotes, try and guess to which of the two books each of them belongs:

“Look for the clutter and prune it ruthlessly. Be grateful for everything you can throw away. Are you hanging on to something useless just because you think it’s beautiful? Simplify, simplify.”

“The point is that you have to strip your work down before you can build it back up. You must know what the essential tools are and what job they were designed to do.”

“Sometimes you will despair of finding the right solution—or any solution. You’ll think, “If I live to be ninety I’ll never get out of this mess.” I’ve often thought it myself. But when I finally do solve the problem it’s because I’m like a surgeon removing his 500th appendix; I’ve been there before.”

“When you find yourself in such a situation, look at the troublesome element and ask, “Do I need it at all?” Probably you don’t. It was trying to do an unnecessary job all along.”

If you guessed On Writing Well four times then well done. The point is that these statements could easily fit into both books. Let’s interpret the interchangeability of these (and many other) statements as proof for the existence of a connection between programming and non-fiction writing.

In the rest of this post I’ll cover McConnell’s advice on three points which are mentioned repeatedly throughout both books—clarity of thought, simplicity, and the importance of iterations.

1. Clarify your thoughts first

Clear minds tend to write clear sentences and produce clear code.

Writers must therefore constantly ask: what am I trying to say? Surprisingly often they don’t know. Then they must look at what they have written and ask: have I said it? Is it clear to someone encountering the subject for the first time? If it’s not, some fuzz has worked its way into the machinery. The clear writer is someone clearheaded enough to see this stuff for what it is: fuzz.

— William Zinsser

More complicated structures require more careful planning, they also benefit from different levels of planning. McConnell says that “from a technical point of view, planning means understanding what you want to build so that you don’t waste money building the wrong thing.“ Investing time into precisely documenting requirements in order to avoid building the wrong features and therefore satisfying the wrong requirements is a form of planning. The same goes for system, object and any other kind of design.

In a sense, planning is a form of clarifying our thoughts. We don’t talk about requirements and create time consuming design documents for their own sake. We design until we feel confident in our ability to get the job done. The point is to plan enough so that a lack of planning doesn’t create major problems later.

Code Complete is all about software construction so the planning McConnell writes about the most is related to the nitty-gritty: how to approach constructing classes and routines from variables, statements and control structures. This is not to say that other levels of planning such as requirements and architecture are less important, in fact, he spends the first part of the book talking about their importance and relation to construction activities.

Let’s take a look at the nitty-gritty.

The Pseudocode Programming Process

McConnell dedicated a whole chapter to this topic. The goal is to solve problems at the level of intent before jumping deep into implementation details. It’s often easier and therefore tempting to start writing code for a routine before clearly stating the problem it’s supposed to solve as well as all of the steps the routine will take. Blindly writing code is a gamble. You are betting your time (and therefore someone’s money) on the code you write to make it into production. This just increases the bond between you and the code which will make abandoning it—after you realize it won’t be needed—more difficult. Before making such bets, improve your chances with the PPP—the Pseudocode Programming Process.

The following may sound very obvious but bear with me for a few sentences. The goal is to think the problem through, identify steps to solve it, and as soon as you are sure that you can implement a certain step (or part of it) just write down a line of pseudocode with the intent of that step (or substep). This saves you the time of actually having to work out the details which is good, because you don’t yet know if this step will make it into production code.

How to Pseudocode:

Use English-like statements to precisely describe operations.
Make it as programming-language-independent as possible.
Keep it at a high enough level to justify its use. Write at the level of intent (what does the operation do instead of the specific steps to do it).
Keep it at a low enough level such that you feel comfortable converting it to production code.

The better and more familiar you are with the language you use and the problem you are solving, the higher level your pseudocode tends to be. A beginner might have to write down the specific steps at first, but if he encounters the same problem multiple times, he will eventually chunk it into one line of pseudocode.

I often struggle with getting the granularity of pseudocode right. Sometimes I write pseudocode that’s detailed to the point where I might as well write code directly. Sometimes—on the other extreme—I write pseudocode on a level that’s too high—this leads me to gloss over problematic parts of the code I later try (and sometimes fail) to write.

Ideally, after converting the problem into actual code you will be able to reuse the pseudocode as comments—avoid redundant comments if the code is clear. This tends to improve readability which will make maintaining and reviewing your code easier.

Keep the above idea—thinking the problem through at the level of intent and only then fully committing to turning your solution into code—in mind while we next take a look at McConnell’s tips for constructing classes and routines.

Tips for Constructing Classes:

Create a general design for the class.
- Define the class’s responsibilities.
- Define what information the class will hide.
- Define exactly what abstraction the class interface will capture.
- Include the last three points as a comment in the source code if possible.
- Make sure that the class’s interface represents a consistent abstraction. (ex. If you offer a findEmployee() routine, it shouldn’t throw an EOFException but an EmployeeNotFoundException)
- Determine whether the class will be derived from another class and whether other classes will be allowed to derive from it.
- Identify key public methods.
- Identify and design nontrivial data structures.
- Minimize accessibility, avoid exposing data and functionality when it’s not necessary to do so.
- Minimize coupling to other classes, avoid depending on code outside of the class as much as practically possible.
- Preserve integrity of the class’s interface and documentation as you modify it.
Construct the routines within the class.
- Follow steps for constructing routines (see below).
Review and test the class as a whole.
- Ideally, each routine is tested as it’s created. After the class starts taking shape it should be reviewed and tested as a whole in order to uncover any issues that can’t be tested at the individual routine level.
Repeat if necessary.
- As most other processes in software engineering, this is by no means a linear process. For example, during construction of the individual routines (step 2), design errors—such as the need for additional routines—might become apparent. If so, go back to designing the class (step 1) before continuing with construction.
- Iterate until you are satisfied.

Tips for Constructing Routines:

Design the routine.
- Clearly define the problem the routine is supposed to solve.
- Name the routine such that the problem it solves is apparent.
- Define information that the routine will hide.
- Define inputs and outputs.
- Define pre- and post-conditions (what is guaranteed to be true before and after the routine is called)
- Think about efficiency but don’t sacrifice readability for dubious performance gains.
- Research available algorithms and data structures, don’t reinvent wheels.
- Summarize the routines job. Use the summary as a comment in the routines header. Ideally, the reader could treat the routine as a black box and only go into the implementation details if necessary.
- Write the pseudocode (level of intent).
Code the routine.
- Convert the pseudocode into actual code.
- Errors in the pseudocode might become more apparent while converting it to actual code. Expect to go back designing the routine (step 1) if you uncover serious errors that impact the whole routine.
Review and test the code and design.
- Mentally check your routine for errors.
- Does the pseudocode fully solve your problem?
- Does the code correspond to the pseudocode?
- Step through your routine with a debugger. This step is so underrated. If you fully understand the routine you just wrote then it shouldn’t take much effort to go through it with a debugger.
- Test your routine.
Repeat if necessary.
- Expect to heavily iterate over the above steps. You will often have to go into the details and implement some pseudocode to validate your approach, then you go back to the pseudocode, then back into implementation details and so on. Just make sure to minimize the time you spend with implementation details. Only implement things to support your reasoning on the pseudocode level, save time and avoid reasoning at the implementation level.
- Iterate until you are satisfied.

Tips for testing routines:

Think about how you will test the routine, both before and as you write it. This tends to result in a modular design and often uncovers errors sooner.
Test all branches of your routine (ex. if you have a switch statement, test all cases).
Boundary analysis, test values +1, -1, and equal to boundaries to avoid off by one errors.
Dirty tests, check if your code fails when it should (too little/much data, invalid data, etc.)
Consider generating random inputs.
Ensure compatibility with old tests if available.

I know, I know. Pre- and post-conditions? Pseudocode? Stepping through with a debugger? For every routine? The above tips sound tedious (they are) and your job is to ship code, that’s fine. The above tips are suggestions to bring more structure into our thought process. Being aware of these optional steps and where they fit into our coding habits is in itself valuable.

2. Keep it simple

Lots of advice specific to writing nonfiction or writing code can be reduced to this: keep it simple.

People at every level are prisoners of the notion that a simple style reflects a simple mind. Actually a simple style is the result of hard work and hard thinking; a muddled style reflects a muddled thinker or a person too arrogant, or too dumb, or too lazy to organize his thoughts.

— William Zinsser

McConnell repeatedly writes that “managing complexity is Software’s Primary Technical Imperative”. At one point he refers to Fred Brook’s No Silver Bullets paper which distinguishes two different types of complexity—essential and accidental. The point is that we should accept only as much complexity as necessary—the essential complexity of the problem at hand. Any rises in difficulty along the path to the final solution should be minimized. In a sense, all advice geared towards improving readability, modularity, maintainability and similar design goals is to increase understanding by reducing complexity. Note that in this post the word complexity refers to intellectual manageability, not computational complexity.

Projects that fail for technical reasons mostly do so because the software is allowed to grow so complex that no one really knows what it does. When a project reaches the point at which no one completely understands the impact that code changes in one area will have on other areas, progress grinds to a halt.

— Steve McConnell

So what can developers do to fight accidental complexity?

Below I list some of the notes I took while reading Code Complete. Each bullet point is my attempt at summarizing a key idea from McConnell’s discussions. Depending on the amount of experience you have, some points will make more sense and some less. The only way to make the most of the advice is to go through the accompanying stories, studies, and code examples in the book. Nonetheless, I’m sure you will find something useful down there.

Treat the following list as a buffet, move on if something doesn’t seem interesting and feel free to pick up and adopt any suggestion you find useful.

Some of McConnell’s advice for reducing complexity:

General
- Before construction, make sure that the groundwork has been laid (problem is well defined, requirements are reasonably stable, architecture is sufficiently well defined, major risks have been addressed etc.)
- „Hide complexity so that your brain doesn’t have to deal with it unless you’re specifically concerned with it. “ Be as restrictive as practically possible when it comes to visibility.
- Limit the negative impact of changes by encapsulating areas that are likely to change such as business rules, hardware dependencies, input and output.
- Make central points of control when possible (ex. put all code related to processing customer payments into one class or subsystem and keep it there). “The reduced-complexity benefit is that the fewer places you have to look for something the easier and safer it will be to change. “
- Assign responsibilities to everything—subsystems, classes, routines, variables, etc—this should help to justify their existence and clarify their usage.
- Use standard techniques whenever possible (widely known algorithms, data structures, design patterns, etc.)
- Consider using brute force. „A brute-force solution that works is better than an elegant solution that doesn’t work. “
- Minimize the amount of knowledge required to make a change. Push unnecessary details to another level so that you can think about them when you want to rather than thinking about all of the details all of the time.
- make code readable from top to bottom
- Quality gates. Set up checks during your project that determine if current work quality is good enough to continue working.
Design
- Form consistent abstractions. Keep the level of abstraction of public interfaces consistent. For example, if a class offers addEmployee() then it shouldn’t offer nextItemInList() but nextEmployee(). Make sure to encapsulate data such that the resulting interface is consistent with the abstraction you want your class to represent.
- Reduce coupling by keeping relations (between subsystems, classes, routines, etc.) small, direct, and flexible.
- Aim for strong cohesion. Code inside of a subsystem, or class, or routine should be strongly related and support a central purpose. (ex. avoid classes that encapsulate unrelated data or behavior).
- Formalize pre- and post-conditions, what must be true in order to use X and what must be true after X finishes its job.
- „Design the interfaces so that changes are limited to the inside of the class and the outside remains unaffected. Any other class using the changed class should be unaware that the change has occurred. “
- Design classes with a high fan-in (ex. your class should be used by a lot of other classes such as a utility class), and a low-to-medium fan-out (ex. your class shouldn’t use and depend on lots of other classes).
- Design for test. Designing such that testing is easy often results in formalized interfaces and decoupled subsystems which is generally beneficial.
- Keep your design modular. Draw diagrams and look at your code as a bunch of black boxes with well-defined interfaces.
- Experimental prototyping. Often you can’t fully define the design problem until you’ve at least partially solved it. In this context, prototyping means writing the absolute minimum amount of throwaway code that’s needed to answer a specific design question.
- Split up validation and work classes. On a class level you could designate the data validation to public classes and let private classes assume that the data they handle is clean. Code outside of the safety zone throws exceptions, code inside the safety zone uses assertions.
Routines:
- Aim for functional cohesion, statements in a routine should all work together to accomplish exactly one job.
- Check pre- and post-conditions.
- Make the name of a routine as short or as long as necessary to make it understandable (other developers should know what the routine does by looking at its name).
- If possible, don’t exceed 200 lines of code, and use no more than 7 parameters.
- Consistently order parameters, group similar ones.
- Avoid using routine parameters as working variables, use local variables in your routine instead.
- Avoid passing parameters to store the output into, return the result.
- Document assumptions as you write the routine (you will forget them later).
- Keep cyclomatic complexity of your routines (number of paths, start with 1 and increment for every if, for, while, repeat, and, or, case…) below ~10.
Loops:
- Comment loop headers, allow the reader to treat loops as black boxes.
- ‎Put control structure related code at the beginning or end of the loop (ex. increment counter indices in a while loop at the beginning or end of its body)
- Loops should be short enough to be viewed all at once, aim for 15-20 lines.
- ‎Loop depth should be at most 3, prepare a good explanation whenever exceeding this limit.
- ‎Avoid altering the loop index in weird ways from inside of the loop.
- Avoid using the loop index outside of the loop
- ‎Give the loop index, its initialization, and end condition meaningful names (avoid i, j, k, etc.).
- Using break or continue (or anything else inside of the loop that alters the control structure) increases the loop’s complexity because the reader has to understand the loop’s body in order to understand the loop. Ideally, the reader should understand how the loop behaves by just looking at the contents of (say) a for loop’s parenthesis.
- Make entry and termination obvious, minimize the number of ways the loop can start and terminate.
Conditionals:
- Outsource complicated tests to routines or variables. Ideally, the reader should understand what a complicated test checks without having to understand the implementation details.
- Show the normal path first, then exceptions. (ex. cover the most common cases first in else-if statements)
- Fully parenthesize expressions (how is a < b == c == d evaluated?)
- ‎Write numeric expressions in number line order, (min < i && i < max) instead of (max > i && min < i).
Variables:
- Initialize constants at the beginning of a program, initialize member variables in constructors.
- Binding time. We differentiate coding time (magic numbers) < compile time (named constants) < load time (read from a file) < object instantiation time (read and set a variable upon object initialization) < just in time (read and set a value every time it is used). After compile time, flexibility increases but so does complexity. The goal is to find a good trade-off based on the project’s requirements.
- Make sure that every variable does exactly one job, never reuse a variable for uncorrelated purposes.
- Avoid implicit meanings (ex. special meaning when an integer variable is negative).
- Encapsulate primitive types in case you expect changes (ex. use a Weight class that internally uses doubles instead of just using doubles)
- Think about rounding errors, division by zero errors, overflows, avoid hard coding data.
Developer Testing:
- Unit, component, integration, regression, system testing.
- Do clean and dirty tests, test if your code works but also test if your code fails when it should.
- Use coverage monitors to ensure high test coverage, choose a good metric such as branch coverage. Most developers are too optimistic when not using tools.
Debugging:
- Understand the problem before trying to fix it.
- Add a unit test that triggers the error and keep it in order to prevent others/yourself from reintroducing the error.
- Don’t ignore compiler warnings, fix and understand all of them to avoid weird problems.
Code Tuning:
- Best preparation for code tuning is writing clean, easy to understand and modifiable code.
- Performance relationships vary across languages, compilers, libraries, machines and versions. Mistrust any general claims about one technique being more efficient than another.
- Always use execution profilers to understand where your program spends its time.
- ‎Only start tuning code if it works correctly. Measure bottlenecks, backup your working code, tune and measure the impact of every change.
Comments:
- Avoid unnecessary comments that explain the obvious. Question the existence of each comment, delete if it’s not helpful (ex. employees.addEmployee(employee); // adds employee).
- Comment data declarations (intent, usage), blocks of code (intent), routines (inputs, outputs, assumptions, limitations, source of algorithms, global effects, source), loops (intent), class headers, etc.
- Only use commenting styles that are easy to maintain (ex. avoid fancy boxes and indentations).
Collaborative Construction:
- Formal inspections, pair programming, walk-throughs, code readings.
- Studies show that formal inspections and pair programming are at least as effective as testing.
- Show your code to others, get feedback, and use it to improve not just your current work but your approach towards future problems.
- ‎Do regression tests, they are essential to producing complex systems.
- Studies show that to maximize defect detection we should combine different testing techniques (formal/informal inspections, prototyping, developer testing, etc.) and views of different people during all stages (requirements, design, construction).
Organizational:
- ‎If you are behind, you likely won’t catch up and it will only get worse. Carefully expand the team, reduce the scope of the project—focus on the most important parts, postpone deadlines.
- Don’t mindlessly apply changes that pop into your head. Store ideas (and requests) for change and deal with them systematically.
- Do daily builds (compile, link, produce an executable, get the code running), and smoke tests (thorough tests of the main features, doesn’t have to be exhaustive).
- Check in code frequently and work in small increments to reduce the amount of integration errors.
- Make quality assurance a part of all development stages. Don’t postpone all testing till the end, it won’t be done properly.
Miscellaneous
- ‎Avoid recursion if it makes you feel uneasy. Only use it as a last resort when an iterative solution is very complex. Think about stack space when using recursion.
- Avoid gotos. There are situations where their use is justified but use them as a last resort. They often hinder compiler optimizations, mistrust efficiency claims.
- You should never have to look at the source code of some class in order to understand how to use it. You should be able to use a class by reading its documentation. Knowing a class’s implementation allows you to exploit it while using the class (often unconsciously). This is error-prone because the class’ developers are only responsible to maintain the interface, implementation details that you assume might change and break your code.
- Be aware of technology waves. Working with technology that is not mature yet means spending a large portion of the day trying to figure out how to use the technology (early wave). Most problems that you face will feel as if you are the first person experiencing them. Working with mature technology means spending more time on building new functionality and less time on understanding the technology since most of the common tasks have been thought of and made easy by other programmers (late wave).
- Use assertions to identify things that should NEVER happen, when an assertion goes off it’s (ideally) not handled but the source code needs to be fixed. Use assertions to verify pre- and post-conditions or to check assumptions in safety zones (ex. private routines) such as checking the range of variables, state of a file/variable, object is not null, size of a data structure meets some criteria, etc.
- „Establish programming conventions (naming, formatting, commenting, etc.) at the start of the project before you begin programming. It’s nearly impossible to change code to match them later. “

Most of this advice is there to keep developers from writing code that’s more complex than it has to be. Whenever I find myself leaning in a little too close towards the screen while working on some “smart” code, I try to lean back for a reality check. Does this have to be difficult? Am I just being silly and making things more difficult than they have to be? Am I using enough hash maps? More often than not I end up ditching the “smart” code and doing it the good old “boring” way.

3. Iterate, iterate, and iterate again

Books, articles, blogposts and non-trivial systems aren’t written in one go. Both authors emphasize the importance of heavily iterating over their work.

Rewriting is the essence of writing well: it’s where the game is won or lost. That idea is hard to accept. We all have an emotional equity in our first draft; we can’t believe that it wasn’t born perfect. But the odds are close to 100 percent that it wasn’t. Most writers don’t initially say what they want to say, or say it as well as they could.

— William Zinsser

The point is not to start with one approach and keep working on it till it’s good enough. The point is to acknowledge that mistakes will be made and learned from while making and abandoning attempts on a best effort basis. McConnell writes: “A first attempt might produce a solution that works, but it’s unlikely to produce the best solution. “

Fun fact, Google rewrites most of their software every few years.

I’ll leave you with McConnell’s emphasis on the importance of an iterative process:

Iteration is appropriate for many software-development activities. During your initial specification of a system, you work with the user through several versions of requirements until you’re sure you agree on them. That’s an iterative process. When you build flexibility into your process by building and delivering a system in several increments, that’s an iterative process. If you use prototyping to develop several alternative solutions quickly and cheaply before crafting the final product, that’s another form of iteration. Iterating on requirements is perhaps as important as any other aspect of the software-development process. Projects fail because they commit themselves to a solution before exploring alternatives. Iteration provides a way to learn about a product before you build it.

— Steve McConnell

Wind Up

I hope that you found some of the tips as useful as I did. Obviously, you won’t remember (and need) all of them, I tried summarizing the ones I think could help most developers. There is much more advice in the book. I would recommend Code Complete to people that have programmed for about a year or two and would like to fill in gaps and get an overview of software construction.

To summarize, we talked about three main issues related to both, nonfiction writing and software construction. First we acknowledged the importance of clarifying thoughts and saw examples of how to structure the class and routine construction processes. Then we took a look at ~~eleventy~~ suggestions on how to keep it simple by avoiding accidental complexity. Finally, we underlined the importance of iterating over and over again until you are satisfied with the outcome.

In case you are interested in more books related to software engineering, McConnell provides a neat reading list at the end of the book. You can also find it online.

Let’s build an audio classifier

2019-05-20T12:08:00+00:00

TL;DR

I use a neural network to build a simple audio classifier and evaluate its performance on the UrbanSound8K dataset.

For this post, our goal is learning how to:

turn sound recordings into feature vectors which a neural network can use,
build a simple classifier with TensoFlow’s estimator API,
run experiments and interpret results.

We will start by taking a look the dataset. Then, we will learn how to extract meaningful information from audio signals to obtain a compact, yet expressive description that the classifier can use. Then, we will learn how to use TensorFlow’s estimator API to build a simple neural network with which we will classify the extracted features. Finally, we will learn how to run experiments, interpret the results, and use what we have learned in future projects.

The Data

In order for the neural network to generalize well we need a sufficiently large amount of varied labeled data. Creating such datasets is challenging as it usually involves first finding quality sources of data, then setting up a system to automate the collection process from various sources, and finally manually labeling the data. Salamon and Bello did just that. Their efforts resulted in the UrbanSound8K dataset, which we will use in this post. I chose this dataset because it is relatively large and easy to work with.

The UrbanSound8K dataset is a collection of 8732 short (~4 second) labeled audio recordings (.wav files) of various urban sounds. Each recording belongs to one of the following 10 classes: air_conditioner, car_horn, children_playing, dog_bark, drilling, enginge_idling, gun_shot, jackhammer, siren, or street_music. The names of the audio files contain various meta-data of which we will use the class id, in other words, the label of each audio recording is contained in the file name. The recordings are conveniently pre-sorted into 10 folds to help with the reproduction and comparison of results.

Feature Extraction

In this section we will turn our audio recordings into feature vectors. We do this because of the network structure we use to build the audio classifier—our data must conform to the input layer’s structure. Hence, we need to turn our audio recordings into arrays of floating point numbers—the feature vectors.

We must also keep in mind that the length of our feature vector is equal to the number of units in the network’s input layer—we therefore have full control over it. However, the cost of training a network (time and compute) usually increases with the number of units in the network. Therefore we will try to keep our feature vectors as small as possible.

A major challenge during the development of audio classifiers is the identification of appropriate content-based features for the representation of the audio recordings. The exact number of different features that people have used up to this point is unknown and irrelevant–the point is that there’s a lot of different ways to concisely represent audio signals, this is a great overview of the most used features. Choosing which feature(s) to extract depends on the nature of the audio sources. Different types of audio sources have different characteristics, the goal is to find features that capture relevant differences between our audio recordings which then the neural network can use during classification.

Our recordings are a kind of environmental sound. An overview of effective features for such audio recordings can be found here. The feature we will use is inspired by the human auditory system and has proven to be very effective—the Mel-frequency cepstrum (MFC). For the purposes of this post we need to know that we extract one MFC per audio recording, that an MFC is made up of Mel-frequency cepstral coefficients (MFCCs), and that an MFC can be stored as a matrix. Each column in an MFCC matrix represents the MFCCs for one frame, and each row represents the extracted MFCCs across all frames (note that some other libraries reverse the columns and rows). So an MFCC matrix is a sequence of MFCCs. In case you are interested how the matrix is computed, see this. Next we will describe how to turn the MFCC matrix into a feature vector by summarizing the extracted sqeuences of MFCCs (rows).

Let’s assume that we extracted all the MFCCs for our audio recordings, now we have one matrix per audio recording which is supposed to be a good representation of the recording’s content. Finally, we will obtain the feature vectors by summarizing each MFCC matrix into one vector. For example, we can summarize one MFCC matrix into a feature vector by computing the mean of every sequence of MFCCs (row in the matrix)—the feature vector would be made up of the means and its length would be equal to the number of MFCCs seqeuences (rows in the matrix). We will go a step further and use multiple summary statistics (minimum, maximum, median, mean, variance, skewness, kurtosis) and obtain the final feature vector by concatenating the vectors we obtained for each of the summary statistics. Therefore, the length of our final feature vector will be the number of coefficients (rows) in our MFCC matrix multiplied by the number of summary statistics we use—in our case we have a feature vector of length 20 * 7 = 140. Note that I chose 20 as the number of coefficients through experimenting with different values.

To extract the features we will use LibROSA—a package for music and audio analysis. Given the path to one of the recordings, we can compute the corresponding MFCC matrix and create our feature vector as follows:

def extract_features_from_file(file_name):
    raw_sound, sample_rate = librosa.load(file_name)

    # one row per extracted coefficient, one column per frame
    mfccs = librosa.feature.mfcc(y=raw_sound, sr=sample_rate, n_mfcc=20)

    mfccs_min = np.min(mfccs, axis=1)  # row-wise summaries
    mfccs_max = np.max(mfccs, axis=1)
    mfccs_median = np.median(mfccs, axis=1)
    mfccs_mean = np.mean(mfccs, axis=1)
    mfccs_variance = np.var(mfccs, axis=1)
    mfccs_skeweness = skew(mfccs, axis=1)
    mfccs_kurtosis = kurtosis(mfccs, axis=1)

    return mfccs_min, mfccs_max, mfccs_median, mfccs_mean, mfccs_variance, mfccs_skeweness, mfccs_kurtosis

Let’s take a closer look at a feature vector we create with the above function. Explicitly, the first feature in our feature vector is the minimum of the first coefficient (first MFCCs sequence) of the recording’s MFCC matrix. The second feature is the minimum value of the second coefficient in the recording’s MFCC matrix. Assume that the MFCC matrix has $k$ rows and start index is $1$, the $k+1$-th feature in our feature vector is the maximum (second summary statistic) value of the first coefficient in the recording’s MFCC matrix. The $2k+1$-th feature is the median of the first coefficient in the recording’s MFCC matrix. And so on for the rest of the summary statistics we chose: the $m$-th summary statistic makes up the $k$ features starting from the $(m-1)*k + 1$-th feature in the feature vector.

The following example shows how to use librosa for performing the feature extraction I described above.

# First we load the audio file.
raw_sound, sample_rate = librosa.load(file_name)  # file must be in the root folder of your project

print("raw_sound:", raw_sound)
print("raw_sound.shape:", raw_sound.shape)
print("\n")

mfccs = librosa.feature.mfcc(y=raw_sound, sr=sample_rate, n_mfcc=20)  # compute the MFCC matrix

# Next we compute the summary statistics, each of them summarizes the MFCC matrix in its own way.
mfccs_min = np.min(mfccs, axis=1)  # row-wise minimum, etc
mfccs_max = np.max(mfccs, axis=1)
mfccs_median = np.median(mfccs, axis=1)
mfccs_mean = np.mean(mfccs, axis=1)
mfccs_variance = np.var(mfccs, axis=1)
mfccs_skeweness = skew(mfccs, axis=1)
mfccs_kurtosis = kurtosis(mfccs, axis=1)

# We obtain the feature vector by concatenating the different summaries.
finalFeatureVector = np.concatenate([mfccs_min, mfccs_max, mfccs_median, mfccs_mean, mfccs_variance, mfccs_skeweness, mfccs_kurtosis])

print("mfccs:", mfccs)
print("mfccs.shape:", mfccs.shape)
print("\n")

print("mfccs_min:", mfccs_min)
print("mfccs_min.shape:", mfccs_min.shape)
print("\n")

print("mfccs_max:", mfccs_max)
print("mfccs_max.shape:", mfccs_max.shape)
print("\n")

print("mfccs_median:", mfccs_median)
print("mfccs_median.shape:", mfccs_median.shape)
print("\n")

print("mfccs_mean:", mfccs_mean)
print("mfccs_mean.shape:", mfccs_mean.shape)
print("\n")

print("mfccs_variance:", mfccs_variance)
print("mfccs_variance.shape:", mfccs_variance.shape)
print("\n")

print("mfccs_skeweness:", mfccs_skeweness)
print("mfccs_skeweness.shape:", mfccs_skeweness.shape)
print("\n")

print("mfccs_kurtosis:", mfccs_kurtosis)
print("mfccs_kurtosis.shape:", mfccs_kurtosis.shape)
print("\n")

print("finalFeatureVector", finalFeatureVector)
print("finalFeatureVector.shape:", finalFeatureVector.shape)

The output of the above example shows the data we are working with during all stages of the feature extraction process I described earlier:

raw_sound: [-0.05454996 -0.13038099 -0.1837227  ...  0.02945998  0.00751382
 -0.03976963]
raw_sound.shape: (88200,)


mfccs: [[-9.78217290e+01 -1.15200264e+02 -1.40122382e+02 ... -1.96356880e+02
  -1.88242525e+02 -1.70682403e+02]
 [ 1.32125684e+02  1.27208858e+02  1.15394368e+02 ...  1.34137467e+02
   1.27680399e+02  1.22533017e+02]
 [-6.74225340e+01 -6.66515689e+01 -5.92767531e+01 ... -3.84596698e+01
  -4.27690513e+01 -4.38633941e+01]
 ...
 [-2.10589355e+00  2.17217352e+00  1.09952991e+01 ...  1.37435825e+01
   1.12389371e+01  1.08770824e+01]
 [-5.70890730e-02  1.68226600e+00  6.01086590e+00 ...  3.57389057e+00
   3.50404748e+00  3.94450867e+00]
 [ 6.35962364e+00  3.37899134e+00  6.57503920e+00 ...  7.80786112e+00
   6.53761178e+00  7.81919162e+00]]
mfccs.shape: (20, 173)


mfccsMin: [-262.1882601   103.68769703  -70.92737613   -7.06636488   -2.57441136
  -14.41774564    3.2479818     5.35776817  -28.04321332   -5.00591745
   -6.77451869  -19.54877245  -13.61704643    1.02649618  -16.86513529
   -2.19287018   -5.88764237   -6.10989292   -6.86591663   -0.98465032]
mfccsMin.shape: (20,)


mfccsMax: [-97.82172897 168.4061548  -28.47245793  38.02142843  21.11869576
  16.15369493  30.22304018  30.83948912   7.8262849   23.61299204
  12.44763379   6.71378187   9.72676893  18.96199983  16.05955284
  24.25329661  23.63521651  21.56967718  14.08734175  18.52652466]
mfccsMax.shape: (20,)


mfccsMedian: [-211.31858874  143.49744481  -54.27675658   20.94282905    9.16580307
   -0.21533032   12.92539716   19.48127578  -15.44112234    8.92052176
    2.95877848   -9.21938462   -3.30998802   12.52612904    5.7662368
   10.03349768    7.50772457    6.01753001    2.9066586    10.10142573]
mfccsMedian.shape: (20,)


mfccsMean: [-2.07550980e+02  1.41899004e+02 -5.35410878e+01  2.05936852e+01
  9.21844173e+00  1.36437309e-01  1.33962109e+01  1.92881863e+01
 -1.49302789e+01  9.21802490e+00  3.25152418e+00 -8.93898900e+00
 -3.07507361e+00  1.17549153e+01  4.85579146e+00  9.71038003e+00
  7.07648620e+00  6.55743907e+00  2.49942263e+00  9.86277491e+00]
mfccsMean.shape: (20,)


mfccsVariance: [889.79984189 268.69156081  88.54314337  75.46131586  25.79757966
  32.35109778  36.47155209  27.3671436   42.67346134  35.26865155
  14.07979971  26.54864055  19.22512523  15.06969567  35.02728229
  20.04404124  21.50347683  29.0669686   15.21941274  14.67586581]
mfccsVariance.shape: (20,)


mfccsSkeweness: [ 0.87052288 -0.47742533  0.35495499 -0.09604224 -0.1575763   0.25362987
  0.50156373 -0.12891151  0.49370992  0.00782589  0.10513811  0.37190028
  0.17411075 -0.45545584 -0.59793439  0.19273781 -0.054743    0.27577162
 -0.07832378 -0.30175746]
mfccsSkeweness.shape: (20,)


mfccsKurtosis: [ 0.90751202 -0.4865782  -0.6830255  -0.64238508 -0.3210699   0.08463533
 -0.26465951 -0.29554489  0.11507486 -0.42076076 -0.32018804 -0.18306213
  0.03231722 -0.42274134  0.1122558   0.02620816  0.24541489 -0.14746111
  0.13099534 -0.15450902]
mfccsKurtosis.shape: (20,)


finalFeatureVector [-2.62188260e+02  1.03687697e+02 -7.09273761e+01 -7.06636488e+00
 -2.57441136e+00 -1.44177456e+01  3.24798180e+00  5.35776817e+00
 -2.80432133e+01 -5.00591745e+00 -6.77451869e+00 -1.95487725e+01
 -1.36170464e+01  1.02649618e+00 -1.68651353e+01 -2.19287018e+00
 -5.88764237e+00 -6.10989292e+00 -6.86591663e+00 -9.84650318e-01
 -9.78217290e+01  1.68406155e+02 -2.84724579e+01  3.80214284e+01
  2.11186958e+01  1.61536949e+01  3.02230402e+01  3.08394891e+01
  7.82628490e+00  2.36129920e+01  1.24476338e+01  6.71378187e+00
  9.72676893e+00  1.89619998e+01  1.60595528e+01  2.42532966e+01
  2.36352165e+01  2.15696772e+01  1.40873418e+01  1.85265247e+01
 -2.11318589e+02  1.43497445e+02 -5.42767566e+01  2.09428290e+01
  9.16580307e+00 -2.15330319e-01  1.29253972e+01  1.94812758e+01
 -1.54411223e+01  8.92052176e+00  2.95877848e+00 -9.21938462e+00
 -3.30998802e+00  1.25261290e+01  5.76623680e+00  1.00334977e+01
  7.50772457e+00  6.01753001e+00  2.90665860e+00  1.01014257e+01
 -2.07550980e+02  1.41899004e+02 -5.35410878e+01  2.05936852e+01
  9.21844173e+00  1.36437309e-01  1.33962109e+01  1.92881863e+01
 -1.49302789e+01  9.21802490e+00  3.25152418e+00 -8.93898900e+00
 -3.07507361e+00  1.17549153e+01  4.85579146e+00  9.71038003e+00
  7.07648620e+00  6.55743907e+00  2.49942263e+00  9.86277491e+00
  8.89799842e+02  2.68691561e+02  8.85431434e+01  7.54613159e+01
  2.57975797e+01  3.23510978e+01  3.64715521e+01  2.73671436e+01
  4.26734613e+01  3.52686516e+01  1.40797997e+01  2.65486406e+01
  1.92251252e+01  1.50696957e+01  3.50272823e+01  2.00440412e+01
  2.15034768e+01  2.90669686e+01  1.52194127e+01  1.46758658e+01
  8.70522876e-01 -4.77425325e-01  3.54954990e-01 -9.60422388e-02
 -1.57576301e-01  2.53629873e-01  5.01563735e-01 -1.28911511e-01
  4.93709923e-01  7.82588861e-03  1.05138109e-01  3.71900277e-01
  1.74110753e-01 -4.55455841e-01 -5.97934394e-01  1.92737805e-01
 -5.47430015e-02  2.75771615e-01 -7.83237770e-02 -3.01757456e-01
  9.07512022e-01 -4.86578202e-01 -6.83025497e-01 -6.42385082e-01
 -3.21069903e-01  8.46353279e-02 -2.64659508e-01 -2.95544886e-01
  1.15074865e-01 -4.20760761e-01 -3.20188037e-01 -1.83062131e-01
  3.23172225e-02 -4.22741337e-01  1.12255804e-01  2.62081612e-02
  2.45414889e-01 -1.47461115e-01  1.30995342e-01 -1.54509024e-01]
finalFeatureVector.shape: (140,)

Note that the feature vectors will potentially vary a lot due to the nature of the recordings (ex. maximum for a gun shot recording will potentially be much larger than for an air conditioner recording). Hence, we will mean normalize the extracted features as follows:

def mean_normalize(featureMatrix):
    mean = np.mean(featureMatrix, axis=0)  # compute mean of each column (feature)
    std = np.std(featureMatrix, axis=0, ddof=1)  # compute sample std of each column (feature)

    featureMatrix -= mean  # subtract each column's mean from every value in the corresponding column
    featureMatrix /= std  # divide values in each column with the corresponding sample std for that column

    return featureMatrix

The feature extraction phase tends to take long—my PC takes about 6 minutes to extract feature vectors of length 140 for one fold of the UrbanSound8K dataset. We will store the extracted features. This way we only need to pay the price of extracting features once, and can reuse them while experimenting during classification later on.

I plan to do a 10-fold cross validation, the following two functions make it easy to extract and store the features systematically. I use extract_feature_from_directories to iterate through the folds and extract features from the recordings they contain–all the extracted feature vectors are combined into one feature matrix (rows are feature vectors, columns are features).

def extract_features_from_directories(parent_dir, sub_dirs, file_ext="*.wav"):
    feature_matrix, labels = np.empty((0, featureVectorLength)), np.empty(0)

    for label, sub_dir in enumerate(sub_dirs):
        for fn in glob.glob(os.path.join(parent_dir, sub_dir, file_ext)):
            try:
                mfccs_min, mfccs_max, mfccs_median, mfccs_mean, mfccs_variance, mfccs_skeweness, mfccs_kurtosis = extract_features_from_file(fn)
                print("Finished processing file: ", fn)
            except Exception as e:
                print("Error while processing file: ", fn)
                continue

            # concatenate extracted features
            new_feature_vector = np.hstack([mfccs_min, mfccs_max, mfccs_median, mfccs_mean, mfccs_variance, mfccs_skeweness, mfccs_kurtosis])

            # add current feature vector as last row in feature matrix
            feature_matrix = np.vstack([feature_matrix, new_feature_vector])

            # extracts label from the file name. Change '\\' to  '/' on Unix systems
            labels = np.append(labels, fn.split('\\')[2].split('-')[1])

    return np.array(feature_matrix), np.array(labels, dtype=np.int)

I use prepare_features to extract and store the training and validation set for one run of the 10-fold cross validation.

def prepare_features(training_dirs, validation_dirs, training_name, validation_name):
    parent_dir = 'Sound-Data'  # name of the directory which contains the recordings
    training_sub_dirs = training_dirs
    validation_sub_dirs = validation_dirs

    # ndarrays
    training_features, training_labels = extract_features_from_directories(parent_dir, training_sub_dirs)
    test_features, test_labels = extract_features_from_directories(parent_dir, validation_sub_dirs)

    # convert ndarray to pandas dataframe
    training_examples = pd.DataFrame(training_features, columns=list(range(1, featureVectorLength+1)))
    # convert ndarray to pandas series
    training_labels = pd.Series(training_labels.tolist())

    # convert ndarray to pandas dataframe
    validation_examples = pd.DataFrame(test_features, columns=list(range(1, featureVectorLength+1)))
    # convert ndarray to pandas series
    validation_labels = pd.Series(test_labels.tolist())

    # store extracted training data
    training_examples.to_pickle('Extracted_Features\\' + training_name + '_features.pkl')
    training_labels.to_pickle('Extracted_Features\\' + training_name + '_labels.pkl')

    # store extracted validation data
    validation_examples.to_pickle('Extracted_Features\\' + validation_name + '_features.pkl')
    validation_labels.to_pickle('Extracted_Features\\' + validation_name + '_labels.pkl')

The following snippet uses the above functions to create the training and validation sets that are used during the first run of the 10-fold cross-validation:

# On the first run I use the first 9 folds for training, the tenth for validation.
training_dirs = ["fold1", "fold2", "fold3", "fold4", "fold5", "fold6", "fold7", "fold8", "fold9"]
validation_dirs = ["fold10"]

# extracts and stores training and validation sets that are used for the first run of the 10-fold cross-validation.
prepare_features(training_dirs, validation_dirs, 'notFold10', 'fold10')

To summarize, we extracted one MFCC matrix per recording. The features we will use during classification are the summary statistics of the MFCC matrix’s coefficients (rows). By concatenating the different summaries of the MFCC matrix we obtain a compact representation of the original audio recording—the final feature vector. The code for this section can be found here.

Next, we will discuss the classification phase and how the neural network will use the feature vectors we developed in this section.

Classification

In this section we will use a neural network to classify the audio recordings. We want to classify each audio recording into one of 10 classes mentioned earlier. Hence, we are dealing with a multi-class classification problem where the classes are mutually exclusive (a recording can belong to only one class). Therefore, we want to build a neural network with a softmax layer at the top.

Our goal is to create a simple but flexible framework which we can use for experiments. We want to control hyperparameters such as the learning rate, the regularization strength, the number of steps the optimization algorithm makes, the batch size, the number of hidden layers, and the number of hidden units in each layer. We also want to leave the doors open for trying out different types of regularization (L1, L2, Dropout), activation functions, to change the number of classes, and to test different optimization algorithms. We want this flexibility while minimizing the number of errors when switching between configurations during experiments. For other issues we are fine with reasonable defaults (set by the TensorFlow’s designers), as long as it allows for enough freedom to quickly try out interesting configurations.

TensorFlow’s Estimator API offers everything we need. Next we will see how to use the estimator API’s DNNClassifier to build our network. The code that follows is a modified version of this.

We have to let our estimator know what type of data it can expect, we do this by defining a feature column. Then, we have to let our estimator know how to fetch data from our dataset—we do this by defining an input function. Finally, we will initialize the classifier and setup the training loop—here we will set up the monitoring of metrics we are interested in while training the classifier (loss curves, confusion matrix).

We start by defining the feature column:

def construct_feature_columns():
    """Construct the TensorFlow Feature Columns.

    Returns:
      A set of feature columns
    """
    return set([tf.feature_column.numeric_column('audioFeatures', shape=featureVectorSize)])

Next we will define two functions for creating the input functions, one for fetching data from the training set, the other for the validation set:

def create_training_input_fn(features, labels, batch_size, num_epochs=None, shuffle=True):
    """A custom input_fn for sending our feature vectors to the estimator for training.

    Args:
      features: The training features.
      labels: The training labels.
      batch_size: Batch size to use during training.

    Returns:
      A function that returns batches of training features and labels during training.
    """

    def _input_fn(num_epochs=num_epochs, shuffle=True):
        idx = np.random.permutation(features.index)
        raw_features = {"audioFeatures": features.reindex(idx)}
        raw_labels = np.array(labels[idx])

        ds = Dataset.from_tensor_slices((raw_features, raw_labels))
        ds = ds.batch(batch_size).repeat(num_epochs)

        if shuffle:
            ds = ds.shuffle(10000)

        # Returns the next batch of data.
        feature_batch, label_batch = ds.make_one_shot_iterator().get_next()
        return feature_batch, label_batch

    return _input_fn


def create_predict_input_fn(features, labels, batch_size):
    """A custom input_fn for sending our feature vectors to the estimator for predictions.

    Args:
      features: The features to base predictions on.
      labels: The labels of the prediction examples.

    Returns:
      A function that returns features and labels for predictions.
    """

    def _input_fn():
        raw_features = {"audioFeatures": features.values}
        raw_labels = np.array(labels)

        ds = Dataset.from_tensor_slices((raw_features, raw_labels))
        ds = ds.batch(batch_size)

        # Returns the next batch of data.
        feature_batch, label_batch = ds.make_one_shot_iterator().get_next()
        return feature_batch, label_batch

    return _input_fn

Next we will define the function that will train the model while periodically printing loss metrics to guide our hyperparameter search during experiments. We will divide the number of training steps into 10 periods. For each period, we train the DNNClassifier for $steps/10$ steps. Then we print loss metrics and continue with the next period. We repeat this 10 times, each time we continue training the DNNClassifier where the previous period left off. The last period gives us the final DNNClassifier which represents our final model.

def train_nn_classification_model(
        learning_rate,
        regularization_strength,
        steps,
        batch_size,
        hidden_units,
        training_examples,
        training_labels,
        validation_examples,
        validation_labels,
        model_Name='no_Name'):
    """Trains a neural network classification model.

    In addition to training, this function also prints training progress information,
    a plot of the training and validation loss over time, as well as a confusion
    matrix.

    Args:
      learning_rate: An `int`, the learning rate to use.
      regularization_strength: A float, the regularization strength.
      steps: A non-zero `int`, the total number of training steps. A training step
        consists of a forward and backward pass using a single batch.
      batch_size: A non-zero `int`, the batch size.
      hidden_units: A `list` of int values, specifying the number of units in each layer.
      training_examples: A `DataFrame` containing the training features.
      training_labels: A `DataFrame` containing the training labels.
      validation_examples: A `DataFrame` containing the validation features.
      validation_labels: A `DataFrame` containing the validation labels.
      model_Name: A `string` containing the model's name which is used when storing the loss curve and confusion
       matrix plots.

    Returns:
      The trained `DNNClassifier` object.
    """

    periods = 10
    steps_per_period = steps / periods

    # Create the input functions.
    predict_training_input_fn = create_predict_input_fn(
        training_examples, training_labels, batch_size)
    predict_validation_input_fn = create_predict_input_fn(
        validation_examples, validation_labels, batch_size)
    training_input_fn = create_training_input_fn(
        training_examples, training_labels, batch_size)

    # Create feature columns.
    feature_columns = construct_feature_columns()

    # Create a DNNClassifier object.
    my_optimizer = tf.train.ProximalAdagradOptimizer(
        learning_rate=learning_rate,
        l2_regularization_strength=regularization_strength  # can be swapped for l1 regularization
    )

    classifier = tf.estimator.DNNClassifier(
        feature_columns=feature_columns,
        n_classes=10,
        hidden_units=hidden_units,
        optimizer=my_optimizer,
        config=tf.contrib.learn.RunConfig(keep_checkpoint_max=1)
    )

    # Train the model, but do so inside a loop so that we can periodically assess loss metrics.
    print("Training model...")
    print("LogLoss error (on validation data):")
    training_errors = []
    validation_errors = []
    for period in range(0, periods):
        # Train the model, starting from the prior state.
        classifier.train(
            input_fn=training_input_fn,
            steps=steps_per_period
        )

        # Use the current model to make predictions on both, the training and validation set.
        training_predictions = list(classifier.predict(input_fn=predict_training_input_fn))
        training_pred_class_id = np.array([item['class_ids'][0] for item in training_predictions])
        training_pred_one_hot = tf.keras.utils.to_categorical(training_pred_class_id, 10)

        validation_predictions = list(classifier.predict(input_fn=predict_validation_input_fn))
        validation_pred_class_id = np.array([item['class_ids'][0] for item in validation_predictions])
        validation_pred_one_hot = tf.keras.utils.to_categorical(validation_pred_class_id, 10)

        # Use predictions to compute training and validation errors.
        training_log_loss = metrics.log_loss(training_labels, training_pred_one_hot)
        validation_log_loss = metrics.log_loss(validation_labels, validation_pred_one_hot)

        # Print validation error of current model.
        print("  period %02d : %0.2f" % (period, validation_log_loss))

        # Store loss metrics so we can plot them later.
        training_errors.append(training_log_loss)
        validation_errors.append(validation_log_loss)

    print("Model training finished.")
    # Remove event files to save disk space.
    _ = map(os.remove, glob.glob(os.path.join(classifier.model_dir, 'events.out.tfevents*')))

    # Compute predictions of final model.
    final_predictions = classifier.predict(input_fn=predict_validation_input_fn)
    final_predictions = np.array([item['class_ids'][0] for item in final_predictions])

    # Evaluate predictions of final model.
    accuracy = metrics.accuracy_score(validation_labels, final_predictions)
    print("Final accuracy (on validation data): %0.2f" % accuracy)

    # Output a graph of loss metrics over periods.
    plt.ylabel("LogLoss")
    plt.xlabel("Periods")
    plt.title("LogLoss vs. Periods")
    plt.plot(training_errors, label="training")
    plt.plot(validation_errors, label="validation")
    plt.legend()
    # plt.show()  # blocks execution
    plt.savefig('Results\\' + model_Name + '_loss_curve.png', bbox_inches='tight')
    plt.gcf().clear()

    # Create a confusion matrix.
    cm = metrics.confusion_matrix(validation_labels, final_predictions)

    # Normalize the confusion matrix by the number of samples in each class (rows).
    cm_normalized = cm.astype("float") / cm.sum(axis=1)[:, np.newaxis]
    ax = sns.heatmap(cm_normalized, cmap="bone_r")
    ax.set_aspect(1)
    plt.title("Confusion matrix")
    plt.ylabel("True label")
    plt.xlabel("Predicted label")
    # plt.show()  # blocks execution
    plt.savefig('Results\\' + model_Name + '_confusion_matrix.png', bbox_inches='tight')
    plt.gcf().clear()

    return classifier

We train the model by calling the training function we defined above like this:

# unpickle and prepare training data
training_examples = mean_normalize(pd.read_pickle('Extracted_Features\\notFold10_features.pkl'))
training_labels = pd.read_pickle('Extracted_Features\\notFold10_labels.pkl')

# unpickle and prepare validation data
validation_examples = mean_normalize(pd.read_pickle('Extracted_Features\\fold10_features.pkl'))
validation_labels = pd.read_pickle('Extracted_Features\\fold10_labels.pkl')

train_nn_classification_model(
    learning_rate=0.003,
    regularization_strength=1.0,
    steps=5000,
    batch_size=32,
    hidden_units=[120],  # One layer with 120 units, for more layers simply add more integers to the list.
    training_examples=training_examples,
    training_labels=training_labels,
    validation_examples=validation_examples,
    validation_labels=validation_labels)

To summarize, we used the Estimator’s API DNNClassifier to build a simple neural network with which we classified the feature vectors we created earlier. The code for this section can be found here.

Results

In order to estimate how a certain model configuration will perform on unseen data we will do a 10-fold cross-validation using the already provided folds in the UrbanSound8K dataset. My machine takes about 4 minutes (on average across different configurations with 1 layer) to train a DNNClassifier on 9 folds and evaluate it on 1 fold. A 10 fold cross-validation takes my machine about 40 minutes. This severely limits my hyperparameter search, I therefore searched for a good model configuration while training on 9 folds and validating on 1. Then, once I found a configuration which performed well, I ran a 10-fold cross-validation to see how it generalizes.

To perform a 10-fold cross-validation we first have to load the features we extracted and stored earlier. Then we use the training function to train models for the different folds:

def k_fold_cross_validation(training_set_names, validation_set_names):
    """
    Performs a k-fold cross validation. Trains k different models and lets you know how they perform by using the
    corresponding validation set.
    :param training_set_names: List of training sets stored as tuples. Each tuple is a pair of strings, first
    element is the name of the training examples, second element is the name of the corresponding training labels.
    :param validation_set_names: List of validation sets stored as tuples. Each tuple is a pair of strings, first
     element is the name of the validation examples, second element is the name of the corresponding validation labels.
    """

    # group each training set with its corresponding validation set
    folds = zip(training_set_names, validation_set_names)

    for (training_name, validation_name) in folds:

        training_examples, training_labels = load_features(training_name)
        validation_examples, validation_labels = load_features(validation_name)

        print("#####################################################################################")
        print("Model is trained with ", training_name[0], "and validated with", validation_name[0])
        train_nn_classification_model(
            learning_rate=0.003,
            regularization_strength=0.1,
            steps=5000,
            batch_size=32,
            hidden_units=[120],
            training_examples=training_examples,
            training_labels=training_labels,
            validation_examples=validation_examples,
            validation_labels=validation_labels,
            model_Name=training_name[0])
 

def load_features(dataset_name):
    """
    Unpickles the given examples and labels. Mean normalizes the examples.
    :param dataset_name: Pair of names referring to an example and corresponding label set.
    :return: Actual dataset as a pair, first element are the mean normalized examples (pandas DataFrame), second
     element are the corresponding labels (pandas Series).
    """

    examples_path = 'Extracted_Features\\' + dataset_name[0]
    # unpickles and mean normalizes examples
    examples = mean_normalize(pd.read_pickle(examples_path))

    # unpickles labels
    labels_path = 'Extracted_Features\\' + dataset_name[1]
    labels = pd.read_pickle(labels_path)

    return examples, labels

Finally, we perform the 10-fold cross validation by specifying the folds and calling the above k_fold_cross_validation function. Note that the k is equal to the number of pairs in training_set_names and validation_set_names. Further, the first cross validation run will use the first tuple in training_set_names (corresponds to the first 9 folds) for training, and the first tuple in validation_set_names (corresponds to the tenth fold) for validation. Similarly for the rest of the runs.

# order in training_set_names matches the order in validation_set_names
training_set_names = [('notFold1_features.pkl', 'notFold1_labels.pkl'),
                      ('notFold2_features.pkl', 'notFold2_labels.pkl'),
                      ('notFold3_features.pkl', 'notFold3_labels.pkl'),
                      ('notFold4_features.pkl', 'notFold4_labels.pkl'),
                      ('notFold5_features.pkl', 'notFold5_labels.pkl'),
                      ('notFold6_features.pkl', 'notFold6_labels.pkl'),
                      ('notFold7_features.pkl', 'notFold7_labels.pkl'),
                      ('notFold8_features.pkl', 'notFold8_labels.pkl'),
                      ('notFold9_features.pkl', 'notFold9_labels.pkl'),
                      ('notFold10_features.pkl', 'notFold10_labels.pkl')]

validation_set_names = [('fold1_features.pkl', 'fold1_labels.pkl'),
                        ('fold2_features.pkl', 'fold2_labels.pkl'),
                        ('fold3_features.pkl', 'fold3_labels.pkl'), 
                        ('fold4_features.pkl', 'fold4_labels.pkl'),
                        ('fold5_features.pkl', 'fold5_labels.pkl'), 
                        ('fold6_features.pkl', 'fold6_labels.pkl'),
                        ('fold7_features.pkl', 'fold7_labels.pkl'), 
                        ('fold8_features.pkl', 'fold8_labels.pkl'),
                        ('fold9_features.pkl', 'fold9_labels.pkl'),
                        ('fold10_features.pkl', 'fold10_labels.pkl')]


k_fold_cross_validation(training_set_names, validation_set_names)

In my (fairly limited) experiments, the best configuration I found achieved an average accuracy of 70.3% across all 10 folds:

classifier = train_nn_classification_model(
    learning_rate=0.003,
    regularization_strength=0.2,
    steps=10000,
    batch_size=32,
    hidden_units=[120],
    training_examples=training_examples,
    training_labels=training_labels,
    validation_examples=validation_examples,
    validation_labels=validation_labels,
    model_Name=training_name[0])

The loss curves and confusion matrices are similar across folds. The following is obtained by training on the first 9 folds and validating on the 10th:

While trying to compare results I found only one article that used simple fully connected neural networks to classify the UrbanSound8K dataset. Their best classification accuracy is 72.2% using 2 hidden layers and 3000 units per layer.

Note that there are much better approaches for classifying the UrbanSound8K dataset. I used a simple fully connected neural network with one layer and few hidden units due to hardware constraints. The most common deep learning based approach for classification of sounds is to convert the audio file to an image (ex. spectrogram, MFCC, CRP), and then use a convolutional neural network to classify the image. The best classification accuracy on the UrbanSound8K dataset I could find is 93%, the approach is described here.

The approach I used in this post is by no means a good one when it comes to maximizing classification performance. It’s good for learning because it can easily be transferred to similar problems to get working prototypes quickly. In case you wish to use a different dataset of audio recordings you could research what features work well for your data and adapt the feature extraction process. In case you use fewer or more classes, simply change the n_classes parameter. In case you want to try out more layers and/or units, simply change the hidden_units parameter. You can swap l2_regularization_strength for l1_regularization_strength or add dropout=0.5 to the initialization phase of the DNNClassifier. You can swap the proximalAdagradOptimizer with another optimizer (ex. tf.train.AdagradOptimizer(learning_rate=learning_rate)).

To conclude, we learned how to use librosa to extract features from audio files. Then we learned how to build a simple but flexible framework for quick experiments with the Estimator API. Finally we learned how to evaluate results and compared it with the results of others. Feel free to comment on what I could have done better or what you would have done differently.

Computing Visibility Polygons

2019-02-20T12:34:00+00:00

TL;DR

I discuss the gist and limitations of my implementation of Joe and Simpson’s visibility polygon algortihm, that is, an asymptotically optimal algorithm for computing the visibility polygon from a point inside of a simple polygon.

The Problem

Recently, I took a course on computational geometry and got interested in the notion of visibility. Besides its applications in hidden surface removal (HSR) algorithms and exact robot motion planning, it received a great amount of attention through the art gallery problem. The first application I could think of when thinking about visibility were video games that incorporated a fog of war, that is, the computation of a player’s surroundings that is visible to him as he navigates a map. Unfortunately, the algorithm we will be talking about doesn’t work with obstacles, making it of limited use in games.

There are many different types of visibility problems, some deal with finding viewpoints to illuminate certain parts of the environment, while others deal with computing visible parts of the environment for a given viewpoint. Some of them consider a single viewpoint, some consider multiple viewpoints, and yet others consider different types of view elements than a point—the list goes on and on. In this post we will focus our attention on the following variant.

Problem Statement. Given a viewpoint $z$ inside of a simple polygon $P$ with $n$ vertices, we want to compute the visibility polygon $VP(P, z)$, which consists of all points in $P$ visible from the viewpoint $z$. We say that point $p$ is visible from point $q$ (and conversely, $q$ is visible from $p$) if and only if the line segment $\overline{pq}$ lies completely in $P$.

Background

The visibility polygon from a single viewpoint $z$ can be computed naively in $\mathcal{O}(n^2)$ time. Simply cast a ray from $z$ towards every vertex of the polygon $P$ in, let’s say, counter-clockwise order. For each ray we iterate through all edges and store the closest (Euclidian distance to $z$) intersection as a vertex of the visibility polygon. Correctness follows from the observation that the visibility polygon’s boundary changes its shape only due to vertices of the polygon.

The above approach is simple yet computationally inefficient for large polygons. Fortunately, more efficient algorithms have been published. For example Asano’s $\mathcal{O}(nlogn)$ time sweeping algorithm and Joe and Simpson’s $\mathcal{O}(n)$ time algorithm (yes, those are the ones used by CGAL).

Quick Background of Joe and Simpson’s algorithm. Linear time algorithms have been shown to be optimal for computing the visibility polygon from a single viewpoint inside of a simple polygon. Such an algorithm was first proposed by ElGindy and Avis, it requires three stacks and is quite complicated. Then, a conceptually simpler algorithm requiring only one stack was proposed by Lee. Later, Joe and Simpson showed that both algorithms return wrong results for polygons that wind sufficiently, they published a correction of Lee’s algorithm. It is their algorithm that we’ll take a look at.

The Algorithm

Instead of a futile attempt to try and capture the algorithm in more detail than Joe & Simpson did, I’ll present to you an overview that will be enough to understand the main idea. We’ll also take a closer look at parts of the algorithm for which there is no pseudocode in the paper—the pre- and post-processing.

The algorithm runs in $\mathcal{O}(n)$ time and space. It makes assumptions on the input which we establish in the preprocessing step—this produces a list of vertices $V = v_0, v_1, \cdots, v_n $ that represents the boundary of $P$ in a specific order, depending on the position of viewpoint $z$ as described in the paper (section two, first two paragraphs).

Preprocessing

The preprocessing will shift the polygon such that the viewpoint $z$ becomes the new origin and it will rotate the polygon such that the closest vertex $v_0$ lies on the x-axis next to $z$—this ensures a line of sight between the first vertex in $V$ and our point of reference $z$ which makes the algorithm’s subsequent design simpler. We’ll make rotations simpler by working with coordinates in polar form.

	private static Pair<VsRep, Double> preprocess(CCWPolygon pol, Point2D z) {
		// shift polygon such that z is origin
		pol = pol.shiftToOrigin(z);
		
		// determines the closest vertex to z
		boolean zIsVertex = pol.vertices.contains(CommonUtils.origin2D);
		PolarPoint2D v0 = getInitialVertex(pol, zIsVertex);
		
		// converts the polygon's vertices from Cartesian to polar
		List<PolarPoint2D> V = pol.vertices.stream().map(x -> new PolarPoint2D(x)).collect(Collectors.toList());
		
		// adjusts list V such that v0 is at the beginning
		placeV0First(V, v0);
		
		// if z is on boundary then [v0, v1, ..., vk, z] -> [z, v0, v1, ..., vk]
		adjustPositionOfz(V, zIsVertex, z);
		
		// rotate all points of the shifted polygon clockwise such that v0 lies on the x axis
		for (PolarPoint2D curr : V) {
			if (!curr.isOrigin())
				curr.rotateClockWise(v0.theta);
		}
		// return the preprocessed vertices and the angle of rotation
		return new Pair(new VsRep(V, zIsVertex), v0.theta);
	}

Main Idea

Then the algorithm proceeds towards its three main procedures—Advance, Retard, and Scan—that handle the three different scenarios that can occur during the monotone scan of $V$—that is, of $P’s$ boundary. While iterating through $V$, a stack $S = s_0, s_1, \cdots, s_t$ of vertices is maintained—it represents a possible subset of vertices of the final visibility polygon. Note that vertices in $S$ are not necessarily vertices of the final visibility polygon. The three procedures are responsible for handling vertices while iterating through $V$ such that the final content of $S$ is all the information necessary for the postprocessing step to construct the final visibility polygon. Advance is responsible for pushing vertices from $V$ onto the stack $S$, retard for popping vertices from the $S$, and scan for skipping vertices in $V$ that have no business modifying $S$. Assume that $V$ is being scanned with $v_j, v_{j+1}$ as the current edge, and that $v_j$ is visible—for $v_{j+1}$ one of the following three cases can occur:

$v_{j+1}$ is visible so that the newly discovered edge doesn’t obstruct previous vertices $\implies$ Advance is called—it pushes $v_{j+1}$ onto $S$.
$v_{j+1}$ is visible and the newly discovered edge obstructs previous vertices $\implies$ Retard is called—it pops obstructed vertices from $S$ and pushes $v_{j+1}$.
$v_{j+1}$ is invisible, it’s obstructed by previous vertices $\implies$ Scan is called—it doesn’t modify $S$ and keeps iterating through $V$ until it reaches a visible vertex.

The algorithm switches between the three procedures until $V$ is scanned completely—at this point $S$ contains all the necessary information to construct the final visibility polygon. The switching between procedures depends on the cumulative angular displacement of scanned vertices with respect to the viewpoint $z$—it is modified upon handling a new vertex from $V$, exactly how is described in the paper.

Postprocessing

The preprocessing step modified the original input polygon $P$ in order to establish assumptions made by the rest of the algorithm—mainly that the first vertex $v_{0}$ that we process is visible which eliminates the need for subsequent special cases. The main algorithm therefore worked on a modified input polygon $P’$ and viewpoint $z’$ and therefore produced $VP(P’, z’)$ whose coordinates correspond to $P’$ and not to our original input polygon $P$.

The postprocessing converts $VP(P’, z’)$ to the corresponding visibility polygon in $P$ by taking $VP(P’, z’)$ and applying “inverse” operations to the ones applied to $P$ and $z$ during the preprocessing. Before shifting and rotating $VP(P’, z’)$ to obtain $VP(P, z)$ we have to reverse the stack’s content to establish counterclockwise order—we pushed vertices onto the stack while iterating in counterclockwise order, just popping them would give us $VP(P’, z’)$ in clockwise order.

	private static CCWPolygon postprocess(List<VertDispl> pre_s, VsRep V, Point2D z, double initAngle) {
		if (V.zIsVertex)
			pre_s.add(0, new VertDispl(new PolarPoint2D(CommonUtils.origin2D), 0));
		
		// reverse order of stack to establish CCW order of final visibility polygon
		Collections.reverse(pre_s);
		
		// convert VertDispl to PolarPoint2D
		List<PolarPoint2D> rotatedPol = pre_s.stream().map(v -> v.p).collect(Collectors.toList());
		
		// rotates points back to original position before the rotation in preprocess()
		for (PolarPoint2D curr : rotatedPol) {
			curr.rotateClockWise(-initAngle);
		}
		
		// convert PolarPoint2D to Point2D
		List<Point2D> shiftedPol = rotatedPol.stream().map(v -> v.toCartesian()).collect(Collectors.toList());
		
		// shifts points back to their position before the shift in preprocess()
		for (Point2D curr : shiftedPol)
			curr.setLocation(curr.getX() + z.getX(), curr.getY() + z.getY());
		
		return new CCWPolygon(shiftedPol);
	}

Time and space complexity. Each vertex is scanned just once, at most two vertices are pushed onto the stack $S$ at each iteration, and popped vertices are never pushed again. This implies that $V$ and $S$ contain a linear number of vertices with respect to the input polygon’s size $n$. Hence, the algorithm runs in $\mathcal{O}(n)$ time and space.

What I left out. The degenerate cases and intricacies regarding the angular displacement and how this affects the switching between advance, scan, and retard with a detailed example on how exactly this algorithm avoids errors made by previously published algorithms is contained in the paper. It also contains the rationale that justifies the use of angular displacements and of course a proof of the algorithm’s correctness.

Tests

I included tests of the pre- and post-processing steps, see TestPreprocessing.java and TestVisibilityPol.java for details. All of the following visualizations can be reproduced via the DrawVisibilityPolygons.java file.

Given the input polygon $P$ and a viewpoint $z$ I created tests for the following six scenarios. The polygon can be either convex or concave, for both types $z$ can be in $P$’s interior, or on an edge of $P$’s boundary, or on one of $P$’s vertices. The following figures are visualizations of those six scenarios.

The algorithm can also be used to create the illusion of computing the visibility region from multiple viewpoints, that is, all points in $P$ visible from at least one of the viewpoints. I say illusion because the algorithm actually computes the visibility polygon for each of the viewpoints individually and unions them by drawing them onto the same plane. This points to a natural approach to actually compute the visibility region, we could use Joe and Simpson’s algorithm to compute the visibility polygons individually and union them with, say, Martinez et al.’s algorithm.

Usage

The VisibilityPolygon class can be used to compute the visibility polygon from a point inside of a simple polygon (given as n vertices in counterclockwise order) in O(n) time and space. Here is an example:

    // initialize polygon vertices in CCW order
	List<Point2D> vertices = new ArrayList<>();
	vertices.add(new Point2D.Double(-2, 2));
	vertices.add(new Point2D.Double(6, 2));
	vertices.add(new Point2D.Double(4, 6));
	vertices.add(new Point2D.Double(1, 4));
	vertices.add(new Point2D.Double(-1, 6));
	vertices.add(new Point2D.Double(-2, 4));
	
	// initialize polygon
	CCWPolygon pol = new CCWPolygon(vertices);
	
	// initialize viewpoint
	Point2D z = new Point2D.Double(4, 4);
	
	// VP contains the visibility polygon from z in pol in CCW order.
	CCWPolygon VP = VisibilityPolygon.computeVisPol(pol, z);

Robustness Issues

Substituting floating-point arithmetic for the real arithmetic assumed in the paper doesn’t go unpunished. My implementation will fail for certain inputs due to round-off errors cause by the inherent limitations of floating-point arithmetic. A straightforward solution would be to make use of some library that allows for arbitrary-precision arithmetic such as Apfloat or JScience.

Another approach for obtaining a less robust but presumably more efficient implementation would be to modify the predicates and experimentally assess the improvement in robustness. In order to implement the above algorithm in a robust manner, it is necessary to robustly implement the predicates upon which it relies. The algorithm repeatedly runs a two dimensional orientation test to determine whether a point lies to the left of, to the right of, or on a line defined by two other points. It also computes intersections between lines and segments, half-lines and segments, and between two segments. We will see that all the predicates can be reduced to orientation tests.

For the orientation test we use the determinant approach—it’s fast and immediately applicable to double precision floating-point inputs.

The orientation test is performed by evaluating the sign of $orientation(A, B, C)$:

\begin{equation} orientation(A, B, C) = \begin{vmatrix} a_x & a_y & 1 \\ b_x & b_y & 1 \
c_x & c_y & 1 \end{vmatrix} = \begin{vmatrix} a_x - c_x & a_y - c_y \\ b_x - c_x & b_y - c_y \end{vmatrix} \end{equation}

If $orientation(A, B, C)$ is less than 0 then $C$ lies to the right of the line that goes through $A$ and $B$, if greater than 0 then $C$ is to the left of, and if equal to 0 then $C$ lies on the line.

Next we’ll take a look at the connection between the orientation test and other geometric predicates that my implementation uses. The problem of testing whether a line and a segment intersect can be reduced to two orientation tests. To test whether a line $l$ and a line segment $ls$ intersect we test whether an endpoint of $ls$ lies on $l$ or whether the interior of $ls$ intersects $l$. For the former we simply test whether endpoints of $ls$ lie on $l$ with the line equation in point-slope form. Testing whether the interior of $ls$ intersects $l$ is equivalent to testing whether the endpoints of $ls$ lie on opposite sides of $l$—can be determined with two orientation tests. The other tests can be reduced similarly—they have more special cases, see the provided implementation for details. The fact that all predicates are reducible to orientation tests makes me believe that implementing the orientation test in a robust manner could significantly improve the algorithm’s robustness.

At this point I’ll quote Schirra’s advice that is directly applicable to our problem:

The straightforward approach to implement geometric algorithms reliably is to use exact rational arithmetic instead of inherently imprecise floating-point arithmetic. Unfortunately, this slows down the code by orders of magnitude. As suggested by the exact geometric computation paradigm a better approach is to combine exact rational arithmetic with floating-point filters, e.g. interval arithmetic, in order to save most of the efficiency of floating-point arithmetic for nondegenerate cases. This approach is implemented in the exact geometry kernels of CGAL and LEDA. The use of adaptive predicates `a la Shewchuck is highly recommended.

Therefore, the algorithm’s robustness issues could be partially resolved by replacing the current naive implementation of the orientation test by Shewchuk’s adaptive approach. It would be interesting to compare the impact on robustness and running time when substituting the current orientation test for Shewchuk’s more robust but presumably slower approach. Note that my implementation would still fail after the substitution because of round-off errors—even if all orientation tests are performed flawlessly—due to comparisons between doubles that are sprinkled all over the code.

Summary

In this post we discussed the gist behind Joe and Simpson’s algorithm for computing the visibility polygon from a viewpoint inside of a simple polygon and concluded by taking a look at the robustness issues of my implementation. We saw how to establish assumptions about the input that were made in the paper—the preprocessing step. Then, we discussed the main idea behind the three routines—Advance, Retard and Scan—and in what manner they modify the stack. We took a look at how the final visibility polygon is constructed using the stack’s content left by the three subroutines after the algorithm finished iterating through the polygon’s boundary—the postprocessing step. We concluded with a discussion on robustness issues where we mentioned an approach to resolve them—arbitrary-precision arithmetic —and an idea for future work that makes use of adaptive predicates.

David Glavas

Not so Social GAN

TL;DR

What is Social GAN about?

What’s wrong with Social GAN?

Create your first adversarial examples

TL;DR

What are adversarial examples?

Train the target network

Craft the adversarial examples

References:

On Writing (Code) Well

TL;DR

The Books

1. Clarify your thoughts first

The Pseudocode Programming Process

Tips for Constructing Classes:

Tips for Constructing Routines:

2. Keep it simple

3. Iterate, iterate, and iterate again

Wind Up

Let’s build an audio classifier

TL;DR

The Data

Feature Extraction

Classification

Results

Computing Visibility Polygons

TL;DR

The Problem

Background

The Algorithm

Preprocessing

Main Idea

Postprocessing

Tests

Usage

Robustness Issues

Summary