Henry Cipolla

Tech, gaming, motorsports, and whatever else

Testing multipart/form-data uploads with Post Test Server

The fact that I am getting feature requests means people are actually using my post test server. This makes me happy :) The most recent thing people asked for was multipart/form-data uploads. As an example for this behavior see my test form (which was submitted by a user who was super helpful).

<html>
  <body>
    <form action="http://posttestserver.com/post.php?dir=example" method="post" enctype="multipart/form-data">
      File: <input type="file" name="submitted">
      <input type="hidden" name="someParam" value="someValue"/>
      <input type="submit" value="send">
    </form>
  </body>
</html>

The resulting output should contain information about the file and a link to the actual uploaded file. Remember, this data is public and you should NOT upload private data.

Post Test Server now supports https

Based on feedback, I’ve added an HTTPS certificate to www.posttestserver.com. You may now access the site from: https://posttestserver.com. Hopefully this helps some people.

Simple IP Address Response Server

In order to facilitate some scripts I am writing I just created a service which returns the IP address from which it receives its request. The code for this is a single line of php: echo getenv('REMOTE_ADDR');

Check it out..
Also, please let me know of any other simple services that would be helpful for me to build and host.

Post Test Server now supports custom status messages

Today, the system worked! I received an email from a user asking if I could add support for custom status codes to Post Test Server and I was happy to oblige. So I have added a new parameter which can be passed in the URL called: status_code which causes the server to return a response with the header set to: HTTP/1.0 $status Custom Status (where $status is the value of status_code). Regardless of what code is requested the post will still be dumped in folder in the usual way.

To make it very clear, hitting this url:
http://posttestserver.com/post.php?dir=bot&status_code=650
will cause the server to respond with HTTP/1.0 650 Custom Status.

How to create an EMR job with multiple inputs using the ruby client.

I’m using the ruby client to launch Hadoop jobs on Amazon’s Elastic Map Reduce framework. Things have gone very nicely until I tried scripting a job which draws input from multiple buckets. You can’t use the ‘-input’ option twice and the advice around the internet is to use –args. So, I added:
--args -input,s3n://SomeOtherBucket
to the end my commandline and was displeased to see this option completely omitted from my job.

Grepping the source for ‘–args’ brings up:
commands.parse_options(step_commands + ["--bootstrap-action", "--stream"], [
[ ArgsOption, "--args ARGS", "A command separated list of arguments to pass to the step" ],
[ ArgOption, "--arg ARG", "An argument to pass to the step" ],
[ OptionWithArg, "--step-name STEP_NAME", "Set name for the step", :step_name ],
[ OptionWithArg, "--step-action STEP_ACTION", "Action to take when step finishes. One of CANCEL_AND_WAIT, TERMINATE_JOB_FLOW or CONTINUE", :step_action ],
])

Which shows my problem. –arg and –args are associated with a specific step in the job and as such have to follow the –bootstrap-action or –stream options. They can’t be tacked on to the end.

While on the subject, be wary of using –args because it does not play nicely with commas in the options.

Finally, you need to include –input even if you are also using –arg/–args. Otherwise you get the default wordcount input. So my final command line looked something like:

~/amazon/elastic-mapreduce --create --stream --args -input,"s3n://SomeSecondInput" --enable-debugging --num-instances 16 --master-instance-type c1.xlarge --slave-instance-type c1.xlarge --name "Script Name" --mapper "s3n://MyBucket/map.rb" --reducer "s3n://MyBucket/reduce.rb" --log-uri "s3n://MyBucket/logs" --output "s3n://MyBucket/output" --input "s3n://FirstInput" --bootstrap-action "s3n://MyBucket/bootstrap.sh" --bootstrap-action s3://elasticmapreduce/bootstrap-actions/configurations/latest/memory-intensive

The last bootstrap option is magic, btw.

Fixing colors in the terminal app on OS X Lion

OS X Lion became available to everybody today. Yay. After installing it, I did the usual new-Mac-OS-install-what-broke? check and noticed that as soon as I ssh into any of my boxes I lose all colors in the terminal. Sad. Fortunately the fix is easy. Go to preferences inside the terminal, within the settings tab change “declare terminal as ‘xterm-256color’ to ‘xterm-color’. Things will be happy again.

Macbook Pro: Fix 4SNS/1/40000000 TsOP by replacing Palm Rest Sensor (fixes fan constantly at 100%)

I recently replaced the keyboard in my MacbookPro 2,1 (this is relevant for most older MBPs, not just the 2,1) by following the steps in this great video: http://www.youtube.com/watch?v=jUonF9OSvpA

After I put everything back together the keyboard worked like a champ but after two minutes of uptime my fans would suddenly peg at 6,000 RPM. I tried resetting the SMC controller (http://support.apple.com/kb/ht3964) and when that failed I ran the Hardware Test which I would encourage anybody experiencing hardware troubles to do.

The hardware test came back with: 4SNS/1/40000000 TsOP
I expected to find a nice description of this string online, similar to OBD-2 trouble codes in a car or POST errors provided by any PC manufacturer. To my great dismay this list is either not publicly available, well hidden, or I failed. After some Googling I was able to piece together that the 4SNS indicates a sensor failure and TsOP is the palm rest sensor. Read the bottom of this post for a list of other sensors.

Palm Rest Sensor
If you can open up the laptop and find this sensor on your own then you are more clever than I. After some failure the Internet explained the sensor is built into the keyboard/touchpad connector. I’ve used a picture taken from faqintosh.com to demonstrate the cable:

This connector doubles as the Palm Rest Temperature Sensor

. This connector is usually taped down onto the motherboard and when I peeled the tape off I accidentally pulled a little bit of the sensor off of it. Resulting in this situation:

The temp sensor has been ripped off the connector

.

To fix this, the entre connector needs to be replaced. To do this remove the palm rest + keyboard assembly (use the steps in the video above). You will likely have to break some Warranty tape. However, if your laptop is still under warranty you probably wouldn’t be reading this to begin with. Make note of the keyboard backlight connector.

Remember to reconnect this.

If you fail to reconnect this the keyboard backlight won’t work which is probably not so bad but OS X will detect the keyboard as being a newer mac keyboard with different function key layout. So volume up will go from f5 to f12! This mistake had me scratching my head for quite some time.

The final area of caution is the connectors for the power button and magnetic lid sensor. It is glued on to the underside of the speaker grille and shouldn’t be ripped off because it will make installing the new one harder. Instead, pull it up while sliding something under it to break the glue seal.

Be gentle here to avoid damaging the plastic?/rubber? underneath the grille

Installation is the opposite of removal and none of the pictures I took were especially interesting.

Hope this helps.

List of Other Sensors (taken from this list: http://forums.appleinsider.com/showthread.php?t=114028)

  • TB0T Battery TS_MAX Temp
  • TB1T Battery TS1 Temp
  • TB2T Battery TS2 Temp
  • TB3T Battery Temp
  • TC0D CPU 0 Die Temp
  • TC0P CPU 0 Proximity Temp
  • TG0D GPU Die – Digital
  • TG0P GPU 0 Proximity Temp
  • TG0T GPU 0 Die – Analog Temp
  • TG0H Left Heat Pipe/Fin Stack Proximity Temp
  • TG1H Left Heat Pipe/Fin Stack Proximity Temp
  • TN0P MCP Proximity
  • TN0D MCP Digital
  • Th2H Right Fin Stack Proximity Temp
  • Tm0P Battery Charger Proximity Temp
  • Ts0P Palm Rest Temp

Added features to my free HTTP Post Dumping / Testing service.

A while ago I put together a very simple php script that dumps any HTTP Post it receives:

To my great surprise, people have actually started using it! I also found myself using it more and more in my own debugging and so I have added two features which further my original mission of maximizing this project’s value to effort ratio. Total effort is still under one hour.

New Features:

  1. Now dumps all Header parameters. Previously I was only dumping the ones that I thought relevant. Now you can see carrier and ISP inserted fields as well.
  2. You may now specify a directory in the query string to have your post written there. So instead of hitting: http://posttestserver.com/post.php you instead hit: http://posttestserver.com/post.php?dir=myself and then after selecting the current date you will see a myself directory containing your uploads

Hopefully this continues to help people. Even if it doesn’t, it helps me so I’m satisfied.

0-indexed numbering makes more sense in programming than 1-indexing. (Dijkstra agrees)

Let me present two lists of programming languages.

List One List Two
COBOL (1959) BASIC (1964)
Fortran (1957) C (1973)
FoxPro (?) C# (2001)
Lua (1993) Java (1995)
MATLAB (late 1970s) PHP (1995)
PL/I (1964) Python (1991)
RPG (1959) Ruby (1995)
Smalltalk (1972)

Complete list on Wikipedia

Unfortunately SEO dictates that the title of this post already describes these lists: List one is the majority of the 1-indexed programming languages I could dig up and list two is a sampling of 0 indexed languages. You see, last night a good friend of mine asserted that the only reason modern languages use 0 indexed arrays is convention; they are based on old languages which were more hardware-focused. (Chris, feel free to correct me if I’m not phrasing that fairly). Unfortunately (for him), this seems incorrect.

Old languages were 1 indexed

Look at the years in the table above. It seems like 1 indexing was rather popular during the early days of language development. COBOL and Fortran are the among the oldest languages around and thanks to their prevalence in finance, government, and military are still in surprisingly wide use today. But Algol is more fun to focus on. If you aren’t familiar with language history, look at the Wikipedia entry on languages derived from Algol and notice how every language in list 2 (the 0-indexed list) is derived from ALGOL (a 1-indexed language).

Clearly, the convention early on was 1-indexing which is not surprising. Up until this point people tended to think of 1 as the first item in a set. But as languages evolved, there came later was a barrage of 0-indexed languages.

Dijkstra’s opinion

If you are unfamilar with Edsger Dijstra then: a) you make me sad and b) do some quick googling. Dijkstra is credited for inventing many of the core concepts of software development, including abstraction, using loops, Dijkstra’s algorithm (shortest-path), and much more. He even wrote a paper on this matter: which is based on the premise that in programming we often have to operate on a sequence of natural numbers. If you disagree with that, stop reading. His opinion explained:

Start with the sequence of numbers 2 through 12. (2,3,4,5,6,7,8,9,10,11,12). Dijkstra shows the 4 ways to denote these sequences:
a) 2 ≤ i < 13
b) 1 < i ≤ 12
c) 2 ≤ i ≤ 12
d) 1 < i < 13

In the first two options), the difference between the min and the max is equal to the number of values in the range. (13 - 2 == 12 - 1 == 11) and two adjacent number ranges (1,2,3 and 4,5,6) are expressed in a way that the upper bound of the first range will be equal to the lower bound of the second range.

Dijkstra also suggests that in computing it is common to get sequences that consist of only the smallest natural number (0) or sequences that are empty. Sequences starting at 0 would be expressed as -1 < i for methods b and d above. This forces the use nonnatural numbers even though we are only describing natural numbers. With c, an empty set would be represented as: 0 ≤ i ≤ -1 which is hard to support.

So, in Dijkstra’s opinion, a is the only way to cleanly describe sets of natural numbers.

Practical Concerns

Another angle of approach is the examination of what programming in 1-indexed languages looks like:

Array positioning

Consider the common task of copying an array into a specific spot of a larger array. Normally this would look like:

# copy src into dest at position p
copy(src, dest, p) {
for i from 0 to src.length -1
dest[p + i] vect[i]
}

In 1-index land, the extra index needs to be accounted for:

# same thing with 1-indexing:
copy(src, dest, p) {
for i from 1 to src.length
dest[p + i - 1] = vect[i]
}

Hashing

Consider the common task of creating a hash table to optimize lookups. Again, if you don’t believe this is a common task, please stop reading.

To keep it practical, consider a system that keeps track of individual users. Each user has a unique, numeric user ID and a name. This can be represented as:

user {
int id;
string name;
}

To facilitate lookups, we devise a very simple hashing function:

function hash(int input) {
return input % 5;
}

Create an array to store these hashed users
Array user_lookup[5];

Each item in the array is a linked list containing all the users whose ID mapped to that value. So we add users by doing: user_lookup[hash(user.id)].add(user);

However, the first user to come through with id 500 breaks things because their value comes to 0. So now, we have to add a +1 somewhere compensate.

C-style pointer math
One common argument is that in C, arrays are pointers to memory.

So given the following:
int array[5];

The statement: a[3] is actually translated to *(a + 3 * sizeof(int))

Adding accounting for 1 indexing would turn that into:
*(a + 3* sizeof(int) + 1) which gets back to the elegance and unnecessary work arguments suggested above.

Conclusion
Originally, I had hoped to do something clever with structs and unions in C to show that 1 indexing is broken. However, the best I could have achieved with that is proving that it is broken in C which doesn’t actually prove anything about programming a whole. Instead, Dijkstra’s argument on the elegance of 0-indexing coupled with practical examples apply to a much broader range of languages.

Basic Frozen Synapse Level Editing

I recently picked up a copy of Frozen Synapse and have been playing it with my friends. Unfortunately, many years of first having a real job then running a startup has divorced me from video gaming enough that I’m rather terrible at it. So, rather than be defeated I’ve turned my energies into an exploration of the level editing system for a more positive outcome.

At the time of my writing this documentation for the level editor is neither non-existant or very well hidden. As a result, I did some trial and error work and drew a few conclusions about how level editing works which I share below. I also provide images demonstrating many of these points further down. As time goes on I’m sure two things will happen:

  1. The editor (which is currently in beta) will get better
  2. Proper documentation will start existing

As a result, don’t be surprised to find this full of obsolete, inaccurate data. I am not responsible if your computer kills a kitten as a result of inconsistencies between this and reality.

Level Editing Basics:

  • The built in level editor is unstable and laborious. Instead, the optimal way to create levels is to import .pngs with level data within the editor
  • Most graphics editors save as .png. mspaint on windows is a good solution. For the tests below, I used Gimp
  • To import a .png it must have a white background and at least 1 pixel of white all the way around the border. Images with colors on the border will hang the editor.
  • The UI only lets you browse to folders within the Frozen Synapse folder. So I created a new folder inside the psychoff folder named “maps” and put my images in there, so they would be accessible in the game
  • Red pixels make up solid walls, blue pixels make up cover.
  • The colors don’t need to be perfect. I.E. 0xFF0000 is all red, but 0xAA0000 also works
  • 1024×768 is a good starting size for a 3 or 4 man game
  • Circles and diagonals are allowed. But I’d be wary of their impact on pathing
  • A shotgun can aim through a 1 pixel hole or 1 pixel of cover
  • A unit needs at least 23 pixels to make it through an entryway. However the rules are not consistent (i.e. not every 23 pixel entryway works).
  • It is easy to test level pathing in the editor

A blank room to start with

Here is a blank 1024×768 room to get you going:

An empty 1024x768 room for editing

An empty room, ready for editing

An empty, 1024x768 room as it will be rendered with an additional unit for scale.

Rendered version of empty room (with an added unit for scale)

Test 1) Scale

The first thing I wanted to establish, was the scale. How many pixels do I need for a reasonable level? So I created two rooms, one 320×240 one 1024×768. I gave each level a red 6 pixel border (with a slight gradient). Below are the source .pngs and the results as rendered by my game. From this it’s clear that 320×240 is a bit small, 1024×768 is actually quite good, and 6 pixels is a bit much:

Basic 320x240 image

The starting point for most of my tests

Rendered version of basic 320x240 room w/ unit

Rendered version of basic 320x240 room w/ unit

A basic 1024x768 level

A basic 1024x768 level

A basic rendered 1024x768 room w/ unit

A basic rendered 1024x768 room w/ unit

Test 2) Circle

Every map I have seen so far is made up only of corners. After reading Mode 7′s blog post on path finding I would highly recommend avoiding anything other than corners in your maps because as you can tell from the images below, the circle ends up being rendered as a many-edged shape (Priests from Flatland!) which will cause a lot of extra work for the pathing algorithm. That said, I intend to put rounded edges in the first map I make. As a side note, to anybody from Mode7 who ever reads this: Please write more blog posts like the above. I really enjoyed it!

Circle

A circle

A circle rendered by the game

A rendered circle

Test 3) Diagonal Lines And Intersections

For something a bit more reasonable, I tried diagonal lines and a 4 way intersection. Diagonal lines are also something I haven’t seen in any maps and after looking at the results it is pretty clear why. Like the circle diagonal lines cause a very large number of edges which will impact the speed and reliability of the pathing algorithm. You also don’t come across too many diagonal shaped rooms in your day to day life. (Anybody ever been to Microsoft’s main campus? A number of the older buildings are shaped like X’s and contain many diagonal walls. They are very disorienting. So it’s probably best not to use diagonals in that dream house you are building)

Hourglass

An hourglass demonstrating jagged edges and intersections

Rendered hourglass

Rendered Hourglass

Test 4) Color Gradient

This was just for fun. I do not believe there is any reason ever to use any colors other than 0xFF0000 (red) and 0x0000FF (blue). But I was curious how the editor would handle a gradient from red to blue so I created the scenario below. To its credit the result was solid wall in the reds, cover wall in the blues and the area of extreme overlap in the middle was left out. You guys win this round!

gradient test

What happens when we fade from red to blue?

Rendering of a red to blue gradient

The rendered gradient.

Test 5) Small Squares for Scale

Not a test so much as a reference. Below are 7 squares. The is a single pixel, then a 2×2, 3×3, up to 6×6. Then, for no explicable reason I skipped 7×7 and created an 8×8 square with the inside hollowed out. Unsurprisingly a hollowed out square works the same as a regular square. Based on these results it’s pretty clear that pretty much any width wall will work so it’s best to choose what makes the most sense from an aesthetics point of view. If you are trying to give a specific room the feeling of being very strong, make the wall a bit wider. Similarly, tenuous cover zones might want to be made smaller.

Squares for testing scale

Squares ranging from 1 to 8 pixels wide

increasing squares

Squares going from 1x1 to 8x8


Test 6) Minimum Doorway Width

Hopefully somebody who knows more about this than I do will read this and explain to me what I did wrong here. Possibly insulting all that which I hold dear – I deserve it. A common problem with maps I have tried is that units can’t move through doorways because they are too small. The test for this seemed simple enough. I figured the minimum doorway width would be around 25 pixels so I created a bunch of holes of increasing size starting at 20 and ending at 28. I put a unit on one side and double clicked the other side. The result, however was that the auto pathing was only able to go through the 23 pixel hole. I couldn’t even get my unit through the larger holes (hehe) by manually pathing them through it. In a fit of rage I closed off the 23 pixel hole and now the unit just stayed on its side of the level, unable to progress. So the takeways here are curious enough they are worth calling out:

  1. The minimum hole size a unit can go through is not a constant. 23 worked on the same map that 24-28 did not.
  2. It is the responsibility of the map maker to test every doorway. Don’t release a glitchy map.
1024x768 map with holes

Holes ranging from 20 to 27 pixels wide

holes in a wall of increasing size

The strange path between the holes in the wall.

Test 7) Minimum Cover/Hole A Unit Can Fire Through

Spoiler: the answer is 1 pixel. I created a wall and put a single pixel of cover in it. Then a 2 pixel cover, 3, etc. Then put a unit on the other side and decided to see at what point my unit would get shot. Turns out a shotgunner can shoot through a 1pixel opening just fine. There was no delay or any other measurable change between the 1 and larger pixel holes.

Same image as test 6.
Shooting through a 1 pixel hole

Shooting through a 1 pixel hole

Hope this helps

After doing this work, I’m ready to try and create a fun level. It is still a bit odd to be creating a level with an image editor and not some crazy level software (anybody remember the D-Zone level editor for Doom?) but I’m excited to do it. In the future I expect the built in editor will become more stable and quicker to use and it is possible the image importing will fall by the wayside. Until then, I hope this helps!