goto epicfailure;This is my awesome site and blog where I reflect upon hardware, software and the state of the universe. Do not interpret content here as the opinion of my employer. Views and opinions are my own.
https://haaspors.github.io2023-04-18T20:30:02+00:00ESPtool - the magic sflash stub2016-03-03T12:00:00+00:00https://haaspors.github.io/dev/2016/03/03/esptool---the-magic-sflash-stub<p>The open source ESP8266 <a href="https://github.com/themadinventor/esptool">esptool</a> that’s part of the
<a href="https://github.com/pfalcon/esp-open-sdk">esp-open-sdk</a> is used as a tool to create a firmware image
from elf image/binary.
It is also used to flash/program an ESP8266 with this firmware image.
In addition you can do the opposite, download a firmware image.</p>
<h3 id="sflash_stub">SFLASH_STUB</h3>
<p>Browsing through the <code class="language-plaintext highlighter-rouge">esptool.py</code> source code I found an interesting piece of
binary blob I just had to take a closer look at.</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="c1"># Sflash stub: an assembly routine to read from spi flash and send to host
</span><span class="n">SFLASH_STUB</span> <span class="o">=</span> <span class="s">"</span><span class="se">\x80\x3c\x00\x40\x1c\x4b\x00\x40\x21\x11\x00\x40\x00\x80\xfe</span><span class="s">"</span> \
<span class="s">"</span><span class="se">\x3f\xc1\xfb\xff\xd1\xf8\xff\x2d\x0d\x31\xfd\xff\x41\xf7\xff</span><span class="s">"</span> \
<span class="s">"</span><span class="se">\x4a\xdd\x51\xf9\xff\xc0\x05\x00\x21\xf9\xff\x31\xf3\xff\x41</span><span class="s">"</span> \
<span class="s">"</span><span class="se">\xf5\xff\xc0\x04\x00\x0b\xcc\x56\xec\xfd\x06\xff\xff\x00\x00</span><span class="s">"</span></code></pre></figure>
<p>It is used in the following python function.</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="s">""" Read SPI flash """</span>
<span class="k">def</span> <span class="nf">flash_read</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">offset</span><span class="p">,</span> <span class="n">size</span><span class="p">,</span> <span class="n">count</span><span class="o">=</span><span class="mi">1</span><span class="p">):</span>
<span class="c1"># Create a custom stub
</span> <span class="n">stub</span> <span class="o">=</span> <span class="n">struct</span><span class="p">.</span><span class="n">pack</span><span class="p">(</span><span class="s">'<III'</span><span class="p">,</span> <span class="n">offset</span><span class="p">,</span> <span class="n">size</span><span class="p">,</span> <span class="n">count</span><span class="p">)</span> <span class="o">+</span> <span class="bp">self</span><span class="p">.</span><span class="n">SFLASH_STUB</span>
<span class="c1"># Trick ROM to initialize SFlash
</span> <span class="bp">self</span><span class="p">.</span><span class="n">flash_begin</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">)</span>
<span class="c1"># Download stub
</span> <span class="bp">self</span><span class="p">.</span><span class="n">mem_begin</span><span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="n">stub</span><span class="p">),</span> <span class="mi">1</span><span class="p">,</span> <span class="nb">len</span><span class="p">(</span><span class="n">stub</span><span class="p">),</span> <span class="mh">0x40100000</span><span class="p">)</span>
<span class="bp">self</span><span class="p">.</span><span class="n">mem_block</span><span class="p">(</span><span class="n">stub</span><span class="p">,</span> <span class="mi">0</span><span class="p">)</span>
<span class="bp">self</span><span class="p">.</span><span class="n">mem_finish</span><span class="p">(</span><span class="mh">0x4010001c</span><span class="p">)</span></code></pre></figure>
<p>This means the stub is prefixed with 3x 32bit integers, respectively
<code class="language-plaintext highlighter-rouge">offset</code>, <code class="language-plaintext highlighter-rouge">size</code> and <code class="language-plaintext highlighter-rouge">count</code>.
The stub plus prefix is flashed into memory @ <code class="language-plaintext highlighter-rouge">0x40100000</code> and then execution
starts @ <code class="language-plaintext highlighter-rouge">0x4010001c</code> meaning that first 4x 32bit integers of the stub
is probably data.</p>
<h3 id="radare">Radare</h3>
<p>I figured, this was a perfect use case of showing how to use <a href="//rada.re">radare</a>.
Here it is:
<a href="https://asciinema.org/a/a1c9znc34jl8ztm4ien7n2ldm"><img src="/img/r2_esptool_sflash_stub.png" alt="asciinema" /></a></p>
<p>There you have it!
esptool uploads the sflash stub and executes it.
The stub itself contains a small function which calls two functions in a loop.
First <code class="language-plaintext highlighter-rouge">SPIRead</code> is called to fetch a block from flash into RAM.
Then <code class="language-plaintext highlighter-rouge">send_pakcet</code> is called to send this block back over UART.
Both <code class="language-plaintext highlighter-rouge">SPIRead</code> and <code class="language-plaintext highlighter-rouge">send_packet</code> resides in ROM and are so called
ROM functionality.</p>
ESP82662016-03-02T12:00:00+00:00https://haaspors.github.io/hw/2016/03/02/esp8266<p>ESP8266 is a wonderful little piece of hardware I stumbled over some months ago.
It has gotten a lot of attention since some time back in 2014.
Mainly because it is an incredible cheap WiFi module (less than $2 these days).
It’s not until recently I’ve had some time to actually look into it.
Personally I like to know how things work from bottom up, so that means
digging down into electronics and the firmware/software you can either
use out of the box or build/construct yourself for ESP8266.</p>
<p><a href="/img/esp-12e_banana.jpg"><img src="/img/esp-12e_banana.jpg" alt="ESP-12E and a banana for scale" /></a></p>
<h3 id="research">Research</h3>
<p>I early on decided that I wanted to know more about the chip and architecture.
So far I’ve been doing most research on the components of the ESP8266EX version.
The CPU, memory and how it boots.</p>
<h3 id="hardware">Hardware</h3>
<p>ESP8266(EX) is made by <a href="//espressif.com">Espressif</a> and there are multiple vendors out
there selling different kind of modules embedding it.
See <a href="http://www.esp8266.com/wiki/doku.php?id=esp8266-module-family">this list</a> which are somewhat updated.
Personally I’ve bought ESP-01 and ESP-12E and will probably mostly use the latter.
ESP-12E is from AI-thinker based on ESP8266EX chip.
See Espressifs <a href="http://bbs.espressif.com/viewtopic.php?f=51&t=818">newest datasheet</a> (0A-ESP8266EX__Datasheet__EN_V4.7_20160225.pdf)</p>
<h4 id="cpu">CPU</h4>
<p>Espressif is using Tensilica Xtensa in their architecture.
This is a 32bit RISC architecture.
The specific model is <code class="language-plaintext highlighter-rouge">L106</code> also known as <code class="language-plaintext highlighter-rouge">lx106</code>.
As far as I understand the architecture doesn’t have any form for CPU cache.</p>
<p>See <a href="https://github.com/jcmvbkbc/crosstool-NG">Max Filippovs crosstool-ng repo</a> for a C/C++ toolchain.
You will also need some carefully written linker scripts.
An alternative or better option would be to use the <a href="https://github.com/pfalcon/esp-open-sdk">esp-open-sdk</a>
which bundles the toolchain, Xtensa HAL, esptool and the Espressif SDK.
Very handy!</p>
<h4 id="memory">Memory</h4>
<p>There is a lot of different conflicting information on the inter tubes.
The datasheet doesn’t say a lot about it apart from the fact that it
includes SRAM and ROM.
Apparently there are separate data and instruction buses,
respectively dBus and iBus, + a AHB bus.
SRAM nor ROM size is specified explicitly.
However, this is said about size in the datasheet:</p>
<blockquote>
<p>When ESP8266EX is working under the station mode and is connected to the router,
programmable space is accessible to user in heap and data section is 50KB.</p>
</blockquote>
<p>Information gathered from the linker scripts and
looking at how the stack is setup after boot I’ve concluded:</p>
<table>
<thead>
<tr>
<th style="text-align: right">Address</th>
<th style="text-align: center">Size</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align: right">0x3FFE8000</td>
<td style="text-align: center">0x14000</td>
<td>RAM data - Heap</td>
</tr>
<tr>
<td style="text-align: right">0x3FFEC000</td>
<td style="text-align: center">0x4000</td>
<td>RAM data - Stack</td>
</tr>
<tr>
<td style="text-align: right">0x40000000</td>
<td style="text-align: center">0x10000</td>
<td>ROM (incl. bootloader)</td>
</tr>
<tr>
<td style="text-align: right">0x40100000</td>
<td style="text-align: center">0x8000</td>
<td>RAM instructions</td>
</tr>
</tbody>
</table>
<p>So there should atleast be:</p>
<ul>
<li>96kB data RAM</li>
<li>32KB instruction RAM</li>
</ul>
<p>In addition to this, SPI flash can be mapped to memory through a cache of
some sorts.
This is used to read instructions directly from flash instead of copying it
over to instruction RAM first.
Flash cache should be slower than RAM, but I haven’t done any testing myself.</p>
<div class="alert alert-info" role="alert">
I'm probably going to do another post about how memory is organized later.
So please check back later or go find it if it's already there and I have
forgotten to update this section with a link.
</div>
<h4 id="flash">Flash</h4>
<p>By using <a href="https://github.com/themadinventor/esptool">esptool</a> <code class="language-plaintext highlighter-rouge">./esptool.py --port /dev/ttyUSB0 flash_id</code>
my ESP-12E identifies the flash chip:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Manufacturer: e0
Device: 4016
</code></pre></div></div>
<p>Which I believe is BergMicro BG25Q32 (32Mbit - 4MB).</p>
<h3 id="electronics">Electronics</h3>
<p>I’ve skipped all the electronics that you probably either already know or
should read more about elsewhere.
Buying a module like the ESP-12E require you to do a bit of soldering.
The inter tubes are full of articles/posts of how to connect your ESP8266
using a breadboard, protoboard or whatever.
Google it!</p>
<p>One thing I should mention, that made me hurt for a while, is the fact that
when you boot the stock (AT) firmware, ESP8266 draws >200mA.
This means that powering it from USB through your USB to serial adaptor will
probably <strong>NOT</strong> work.
I learned this the hard way;
After a day of cursing and knocking my head against the wall.
You are better of using a separate power supply which can handle 500mA @ 3.3V.
A capacitor or two across VDD/VCC and ground close to the chip is probably
also recommended.
I’m an electronics rookie, so if you are having problems, talk to an expert.</p>
<h3 id="uart">UART</h3>
<div class="alert alert-info" role="alert">
A minor disclaimer!
I'm writing here that baud rate of the ESP8266 is 115200 for most firmware.
People are reporting that they use other baud rates.
So your specific module might be configured differently, depending on the
firmware and build it shipped with or you have flashed/programmed.
</div>
<p>There are a couple of things you should know about the serial interface.</p>
<ol>
<li>The first stage bootloader initializes UART0 with baud rate based on the external crystal oscillator.</li>
<li>The stock (AT) firmware and probably others use a baud rate of 115200.</li>
<li>Booting in UART mode (also known as flashing/programming mode) uses auto sensing baud rate AFAIK.</li>
</ol>
<div class="alert alert-info" role="alert">
The external crystal oscillator is on the modules I have encountered initialized
to 26MHz after cold boot.
This means that the baud rate is 74880.
To get baud rate of 115200 the external crystal oscillator is reconfigured to 40MHz.
Note that if you software reset/warm boot, the first stage bootloader will
not reconfigure the external crystal oscillator. So the baud rate would be
whatever it was before boot. E.g. 115200.
</div>
<p>I’ve experimented with a couple of applications on Linux for serial communication to ESP8266.
I used picocom for a while because it’s pretty straight forward and supports CRLF
which is what you need for executing AT commands using the AT firmware.
Stock <code class="language-plaintext highlighter-rouge">picocom</code> doesn’t support custom baud rates.
You would need to build picocom yourself where you enable it.
So reading bootloader messages wouldn’t work out of the box.
The same thing goes for minicom and screen.
You could probably make it work initializing the serial terminal using <code class="language-plaintext highlighter-rouge">stty</code>.
As I’m no expert on serial terminals, serial communication drivers nor the
kernel infrastructure for this, I’ve just tried to find whats easiest to use.</p>
<p>I highly recommend using <code class="language-plaintext highlighter-rouge">miniterm.py</code> (included in pyserial python package).<br />
<code class="language-plaintext highlighter-rouge">miniterm.py /dev/ttyUSB0 74880</code> for reading the bootloader message and<br />
<code class="language-plaintext highlighter-rouge">miniterm.py /dev/ttyUSB0 115200</code> for anything else unless you want
high speed communication and/or you have firmware that uses something else.</p>
<p>Typical output from the bootloader:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> ets Jan 8 2013,rst cause:2, boot mode:(3,7)
load 0x40100000, len 1396, room 16
tail 4
chksum 0x89
load 0x3ffe8000, len 776, room 4
tail 4
chksum 0xe8
load 0x3ffe8308, len 540, room 4
tail 8
chksum 0xc0
csum 0xc0
2nd boot version : 1.4(b1)
SPI Speed : 40MHz
SPI Mode : DIO
SPI Flash Size & Map: 8Mbit(512KB+512KB)
jump to run user1 @ 1000
</code></pre></div></div>
<p><code class="language-plaintext highlighter-rouge">boot mode:(x,y)</code> gives <code class="language-plaintext highlighter-rouge">x = 3</code> and <code class="language-plaintext highlighter-rouge">y = 7</code> where <code class="language-plaintext highlighter-rouge">x</code> is the boot mode.<br />
<code class="language-plaintext highlighter-rouge">rst cause:z</code> gives <code class="language-plaintext highlighter-rouge">z = 2</code></p>
<table>
<thead>
<tr>
<th style="text-align: center">Code</th>
<th>Cause</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align: center">1</td>
<td>Normal boot</td>
</tr>
<tr>
<td style="text-align: center">2</td>
<td>Reset pin</td>
</tr>
<tr>
<td style="text-align: center">3</td>
<td>Software reset</td>
</tr>
<tr>
<td style="text-align: center">4</td>
<td>Watchdog reset</td>
</tr>
</tbody>
</table>
<div class="alert alert-info" role="alert">
The boot message above says 'SPI Flash Size & Map: 8Mbit'.
This doesn't say anything about the physical size of the SPI flash.
It is just informing that the flash is formatted to contain two 512KB program
areas.
This is a configuration that allows (F)OTA, or what Espressif call cloud update.
It means that you can update the firmware over WiFi.
It also means that there could be two firmware images on the flash and the
ESP8266 itself can boot either of them.
I haven't read up on all the details, but maybe I will in a later post after
I have experimented a bit more with the bootloaders and booting in general.
</div>
<h3 id="boot-modes">Boot modes</h3>
<p>When you reset the ESP8266 it can boot in a couple of different modes.</p>
<table>
<thead>
<tr>
<th style="text-align: center"><code class="language-plaintext highlighter-rouge">MTDO</code></th>
<th style="text-align: center"><code class="language-plaintext highlighter-rouge">GPIO0</code></th>
<th style="text-align: center"><code class="language-plaintext highlighter-rouge">GPIO2</code></th>
<th style="text-align: center">Code</th>
<th style="text-align: left">Mode</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align: center">L</td>
<td style="text-align: center">L</td>
<td style="text-align: center">H</td>
<td style="text-align: center">1</td>
<td style="text-align: left">UART</td>
<td>Download over UART</td>
</tr>
<tr>
<td style="text-align: center">L</td>
<td style="text-align: center">H</td>
<td style="text-align: center">H</td>
<td style="text-align: center">3</td>
<td style="text-align: left">Flash</td>
<td>Boot from SPI Flash</td>
</tr>
<tr>
<td style="text-align: center">H</td>
<td style="text-align: center">x</td>
<td style="text-align: center">x</td>
<td style="text-align: center">4-7</td>
<td style="text-align: left">SDIO</td>
<td>Boot from SD</td>
</tr>
</tbody>
</table>
<p><code class="language-plaintext highlighter-rouge">L</code> = low<br />
<code class="language-plaintext highlighter-rouge">H</code> = high<br />
<code class="language-plaintext highlighter-rouge">x</code> = floating<br />
<code class="language-plaintext highlighter-rouge">MTDO</code> = <code class="language-plaintext highlighter-rouge">GPIO15</code><br />
<code class="language-plaintext highlighter-rouge">MTDO</code>, <code class="language-plaintext highlighter-rouge">GPIO0</code>, <code class="language-plaintext highlighter-rouge">GPIO2</code> (3 bit) forms Code.</p>
<h3 id="whats-next">Whats next?</h3>
<p>I’m going to research a bit more to find out how both Flash and UART boot mode
actually works.
Both modes are already somewhat reverse engineered and documented elsewhere,
but it is fun trying to understand and then explain it in a later post.
So be sure to check back later for more! :)</p>
Contributing to Radare22016-02-26T12:00:00+00:00https://haaspors.github.io/dev/2016/02/26/contributing-to-radare2<p>You might wonder; What’s Radare or Radare2?
Well, head over to <a href="//rada.re">rada.re</a>/<a href="//radare.org">radare.org</a> and check it out.
Basically it is a reverse engineering tool for static software anlysis.
You can also use it for debugging I guess, but my main use of it will be
reverse engineering binary blobs by disassembling machine code to understand
how it works.</p>
<p>Recently I’ve been playing with Tensilica Xtensa processors, and been looking
at a SoC I’ll probably write something about later.
Anyhow; I got hold of the bootloader as a binary blob for this SoC/platform
which I wanted to know how works.
This bootloader is located in the ROM of the device and includes a lot of board
functionality I wanted to understand more about.
So I figured that this was a good time to try out Radare2 finally.
I was a bit worried that Radare2 didn’t support the architecture, but indeed
did it have support for Xtensa ISA using GNU binutils (I think).
Yay!</p>
<p>After using Radare2 for a couple of hours I had found <code class="language-plaintext highlighter-rouge">Visual mode</code> which is a
very interactive way of looking at a binary, but I also discovered that that
some rendering sometimes looked … funny!?
There were some instructions that didn’t have the right color.
Also, I realized that jump/branch instructions didn’t render all lines correctly.
As I was new to Radare at the time it took some time before I got the whole picture.
Radare2 disassembled instructions correctly, but the interpretation was sometimes wrong.
The reason is that the module responsible for analysis (anal) was
very much a <a href="https://github.com/radare/radare2/blob/d17cbff22cee717959b15040bf04f383fca4f1bf/libr/anal/p/anal_xtensa.c">proof of concept implementation</a>.</p>
<p>The first thing I noticed was that the implementation didn’t take care about
signedness of the jump address.
Hence it would in most cases add to the relative address.
Since the jump addresses mostly are signed extended, short jumps backwards
would be interpreted as big jumps forward.
E.g. the 8bit immediate <code class="language-plaintext highlighter-rouge">0xfe</code> would be interpreted as <code class="language-plaintext highlighter-rouge">254</code> instead of <code class="language-plaintext highlighter-rouge">-2</code>.</p>
<p>After some initial hacking I decided to rewrite the Xtensa analysis backend/plugin.
And so I did: <a href="https://github.com/radare/radare2/commit/4d4ab62211d8f84e6b90ae22d099a4186ca5fe46">github.com/radare/radare2@4d4ab622</a>.
The patch(set) got merged today after I finished it last night. Cool!
I had to read most of the Tensilica Xtensa Instruction Set Architecture (ISA) Reference Manual.
Use your friend Google to find it if you are interested.
The instruction set is quite nice, although I found the opcodes somewhat messy
scattered around in terms of functionality grouping.
I mean, probably not horrendous, but identifying all the different branch/conditional jumps
must be done in several steps.
Oh well, it’s not that bad.
I guess, I would be surprised if I only could check a specific bit in the
opcode to figure out that it was a jump.</p>
<p>Here is an asciicast of how it looks right now using git HEAD:
<a href="https://asciinema.org/a/00p5h37dadkauem09wybp0txz"><img src="https://asciinema.org/a/00p5h37dadkauem09wybp0txz.png" alt="asciicast" /></a></p>
Moving over to github pages2016-02-13T22:45:00+00:00https://haaspors.github.io/timeline/2016/02/13/moving-over-to-github-pages<p>It’s weird. A year ago or so I thought using wordpress would be easy and simple
enough to actually write something quite often.
It turns out I was wrong.
First of all I don’t like wordpress for it’s technology.
It’s PHP! :P And uses a database (MySQL).
Second, it is pain to use.
Log in, use a WYSIWYG editor and publish through the browser.
Changing wordpress, extending or touching it apart from installing plugins seems
tedious and waste of time.
Even just hunting for the proper plugins is waste of time if you ask me.
So…</p>
<h3 id="welcome-to-github-pages-using-jekyll">Welcome to <a href="//pages.github.com">github pages</a> using <a href="//www.jekyllrb.com">jekyll</a></h3>
<p>Now this is nice!
I write my posts using markdown in my favorite editor, <code class="language-plaintext highlighter-rouge">git commit</code> and <code class="language-plaintext highlighter-rouge">git push</code>.</p>
<p>The horrible comments plugin I used in wordpress is swapped out with an
incredible simple <a href="//www.disqus.com">disqus</a> integration.
Farewell wordpress comments spam.
You should see my wordpress comments gmail filter.
Seems like spammers are targeting wordpress sites!</p>
<p>I think it looks pretty good.
I’m no designer, but have been part of some UI/UX work in the past.
I’ve spent some time customizing it.
Too much time, but it was all fun and worth it.
It uses <a href="//getbootstrap.com">bootstrap</a> and some javascript for navbar stuff.
Underneath it’s jekyll obviously and bootstrap sass for styling.
Locally I serve it running <code class="language-plaintext highlighter-rouge">bundle exec jekyll serve</code> as you would find in the
github pages documentation.</p>
<p>I strongly recommend my setup, it’s pretty easy, and I especially like the new way of posting!
<a href="//github.com/haaspors/haaspors.github.io">github.com/haaspors/haaspors.github.io</a></p>
Mouse support in tmux 2.12015-10-27T13:38:00+00:00https://haaspors.github.io/tools/2015/10/27/mouse-support-in-tmux-2.1<p>I’m a vim+tmux user/believer!
I recently installed OS X el capitan and updated/upgraded most of my homebrew Cellar to put it like that.
Now, by doing that I got tmux 2.1 where apparently <a href="https://raw.githubusercontent.com/tmux/tmux/master/CHANGES">mouse-mode/mode-mouse is completely rewritten</a>.
I don’t use the mouse for much, but still nice to have when resizing panes.</p>
<p>So, what you need is to change your <code class="language-plaintext highlighter-rouge">.tmux.conf</code></p>
<figure class="highlight"><pre><code class="language-vim" data-lang="vim"><span class="k">set</span> <span class="p">-</span><span class="k">g</span> <span class="k">mode</span><span class="p">-</span><span class="nb">mouse</span> <span class="k">on</span>
<span class="k">set</span> <span class="p">-</span><span class="k">g</span> <span class="nb">mouse</span><span class="p">-</span><span class="k">resize</span><span class="p">-</span>pane <span class="k">on</span>
<span class="k">set</span> <span class="p">-</span><span class="k">g</span> <span class="nb">mouse</span><span class="p">-</span>select<span class="p">-</span>pane <span class="k">on</span>
<span class="k">set</span> <span class="p">-</span><span class="k">g</span> <span class="nb">mouse</span><span class="p">-</span>select<span class="p">-</span><span class="nb">window</span> <span class="k">on</span></code></pre></figure>
<p>to</p>
<figure class="highlight"><pre><code class="language-vim" data-lang="vim"><span class="k">set</span> <span class="p">-</span><span class="k">g</span> <span class="nb">mouse</span> <span class="k">on</span></code></pre></figure>
<p>Short and sweet! :)
My <a href="https://github.com/haaspors/dotfiles/commit/f0a569115c5dbc734ff40b28dbd96d826024cf41">dotfiles github repo</a> is already updated!</p>
eurosportplayer.no2015-10-03T10:42:00+00:00https://haaspors.github.io/apps/2015/10/03/eurosportplayer.no<p>So, last week I signed up for <a href="//eurosportplayer.no">eurosportplayer.no</a> to watch some Europa League fotball (soccer).
The site looks trendy and good, and my experience watching live sports for 90 minutes was overall good.
No HD from what I could find and some occasional hiccups in video streaming.
I can live with that, although HD would be nice!</p>
<p>With that said, the signup process on the other hand needs some constructive criticism!
I’m unsure if they use HTTPS at all.
I didn’t dig deep to find out, but it surely looks like it’s going over HTTP.
I’m probably wrong, but they should let me browse their site over HTTPS by default if you ask me, and that’s not possible at the time of writing.
Anyway, after entering my email address and password I received this email:</p>
<p><a href="/img/Screen_Shot_2015-10-03_at_12_24_06.png"><img src="/img/Screen_Shot_2015-10-03_at_12_24_06.png" alt="email" /></a></p>
<p>Yes, they actually sent me my password back in plain text.
Which probably means they store all passwords in plain text as well.
Come on - it’s 2015, we don’t do these baby mistakes anymore.
Eurosport, Discovery or whoever you contracted; what’s wrong with you people?</p>
<p>As far as I understand, Tv2 bought the rights to English Premier League for the next three seasons, with Discovery Networks Norway as a close contender.
Maybe that’s a good thing afterall!?</p>
Hafslund Android app2015-04-28T20:00:00+00:00https://haaspors.github.io/apps/2015/04/28/hafslund-android-app<p>Some weeks ago I lost my Android phone into a bucket of paint.
It turned out that the phone didn’t quite like that.
I turned it off and gave it a good long shower, - then blow dried it for a 15 minutes.
After some hours I turned it on, which I regret.
I should have waited for a day or two - or even try to extract more of the moisture out of it.
Well, it booted - but I noticed something wasn’t quite right.
So after some minutes it locked up.
Trying to boot the phone now doesn’t work as the kernel seems to be panicking before complete boot.</p>
<p>Anyway, I can boot into recovery and then adb works.
I obviously tried to salvage what was on the SDcard, so I went through the file system.
It was then I made my discovery.
There was a file named <code class="language-plaintext highlighter-rouge">HafslundLog.txt</code> on the SDcard.
In it I found all my received text messages (SMS) recorded.
Even messages I had deleted.
So obviously this was logged to the file when the message arrived to my phone.
I quickly checked the permissions requested by the <a href="https://play.google.com/store/apps/details?id=no.hafslund.energy">Hafslund app</a>.
The current version 1.0.4 did request SMS access. Nooo, Hafslund - you amateurs!</p>
<p><a href="/img/Screenshot-from-2015-04-29-210855.png"><img src="/img/Screenshot-from-2015-04-29-210855.png" alt="hafslund log" /></a></p>
<p>So, a bit annoyed, I fired off an email to Hafslund.
This is unacceptable. What kind of stupid java developers do they employ?
Next day I got a reply from their customer support asking me to elaborate on the matter.
I replied by repeating the first email and mentioning posting this on social media.</p>
<p>One week later I got a new email where Hafslund thanked me for bringing this to their attention.
A new version would be ready over the summer.
Yesterday I got a phone call, again - thanking me, but now they were about to push a new version to the play store in the coming days.
And yes, today, - April 28th 2015 they updated the app.
It no longer needs access to SMS. Yay! Way to go Hafslund.</p>
<p>However, I’m still one Android phone short. Bummer.</p>
G_DECLARE_*_TYPE2015-02-02T12:48:00+00:00https://haaspors.github.io/gobject/2015/02/02/g_declare_*_type<p>Last week <a href="http://blogs.gnome.org/desrt/">Ryan Lortie</a> <a href="http://blogs.gnome.org/desrt/2015/01/27/g_declare_finalderivable_type/">blogged</a> about some new macros I really have been missing for years of doing GObject development.</p>
<p><a href="https://git.gnome.org/browse/glib/commit/?id=3b4cb28e17c6a5dac64eb8afda2b1143757ad7a4">This commit</a> adds <code class="language-plaintext highlighter-rouge">G_DECLARE_{FINAL,DERIVABLE}_TYPE</code> to gtype.h so that some of the GObject boilerplate code could be omitted.
Sweet! It will be available in next version of GLib. Version 2.44.</p>
<p>One thing I would personally like in the macro, as I tend to add to most GStreamer classes, is <code class="language-plaintext highlighter-rouge">{OBJ_NAME}_CAST</code>!
This would do a simple cast without any run-time checks.
Also known as <a href="http://en.wikipedia.org/wiki/Type_punning">type punning</a> according to wikipedia.
Which basically is like <code class="language-plaintext highlighter-rouge">reinterpret_cast</code> in C++.</p>
GDB - run until crash2015-01-21T12:58:00+00:00https://haaspors.github.io/tools/2015/01/21/gdb---run-until-crash<p>When you write code you most likely want to add test coverage for it by adding a unit test, module test or whatever you want to call it.
Personally I mainly work on highly threaded C/C++ code, where there inevitably will be bugs.
Either in the form of non-deterministic tests or in the code it self as deadlocks or data races.
In my current position we have an extensive CI system which from time to time report that there is such a test.
Even though the test runs millions of times each day, it could be months or years between runs that would make the test fail.</p>
<p>I find it very useful to isolate the test and run it through GDB with or without extra debug to find and squash the bug.
Since most of my tests run very fast and then exit, it is nice to run the test over and over again until it either crashes or deadlocks.
This is how I do it:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>GNU gdb (GDB) 7.6.1-ubuntu
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
(gdb) set pagination off
(gdb) break exit
Breakpoint 1 at 0xcafef00dbeef: file exit.c, line 99.
(gdb) commands
Type commands for breakpoint(s) 1, one per line.
End with a line saying just "end".
>run
>end
(gdb) run
</code></pre></div></div>
<p>As you see, the point is to rerun the program when the breakpoint on <code class="language-plaintext highlighter-rouge">exit()</code> occurs.
Also note how I disable pagination, as this will stop the debugged program when output accumulates.</p>
TCP socket - SO_SNDBUF2015-01-13T09:00:00+00:00https://haaspors.github.io/networking/2015/01/13/tcp-socket---so_sndbuf<p>I came across some code that after connecting a non-blcoking TCP socket did</p>
<figure class="highlight"><pre><code class="language-c" data-lang="c"><span class="kt">int</span> <span class="n">sndsize</span> <span class="o">=</span> <span class="mi">2048</span><span class="p">;</span>
<span class="n">setsockopt</span> <span class="p">(</span><span class="n">socket</span><span class="p">,</span> <span class="n">SOL_SOCKET</span><span class="p">,</span> <span class="n">SO_SNDBUF</span><span class="p">,</span> <span class="o">&</span><span class="n">sndsize</span><span class="p">,</span> <span class="k">sizeof</span> <span class="p">(</span><span class="kt">int</span><span class="p">));</span></code></pre></figure>
<p>The socket was then used to send video data. So in other words;
alot of data and it was designed to use as much bandwidth as possible within a maximum threshold.
Since the socket was non-blocking, data got buffered in userspace before delivered to kernel when the socket became writable.
The man page describing <code class="language-plaintext highlighter-rouge">SO_SNDBUF</code> in <code class="language-plaintext highlighter-rouge">man 7 socket</code> states:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>SO_SNDBUF
Sets or gets the maximum socket send buffer in bytes. The ker-
nel doubles this value (to allow space for bookkeeping overhead)
when it is set using setsockopt(), and this doubled value is
returned by getsockopt(). The default value is set by the
wmem_default sysctl and the maximum allowed value is set by the
wmem_max sysctl. The minimum (doubled) value for this option is
2048.
</code></pre></div></div>
<p>Nevertheless, lowering the socket send buffer to 2K actually resulted in an non-optimal transport:</p>
<ul>
<li>Small packets being sent over the connection</li>
<li>Very low bandwidth being utilized</li>
</ul>
<p>Obviously more IO events are generated by the kernel, as calling <code class="language-plaintext highlighter-rouge">send()</code> on the socket quickly would
return <code class="language-plaintext highlighter-rouge">EAGAIN</code>/<code class="language-plaintext highlighter-rouge">EWOULDBLOCK</code>. However, I didn’t expect the kernel to divide TCP segments to a maximum
of 512 bytes TCP data.
After inspecting wireshark and packet timestamps, it would seem like the sender would pause sending for
fractions of a second periodically.
I could see ACKs coming back very fast, but there would be a pause between the ACK received and the
next packet being sent out.
I guess that somehow the kernel and the application couldn’t handle the quantity of ping pongs between
the kernel and userspace due to the low send buffer at the same time as ACKs generated by the receive side.
I haven’t investigated too much, so if you have an explanation, please enlighten me!</p>
<p><img src="/img/Screenshot-from-2015-01-13-094746.png" alt="Wireshark capture" /></p>
<p>Nevertheless, removing the setting of <code class="language-plaintext highlighter-rouge">SO_SNDBUF</code>, and relying on defaults obviously solved the issue.
Looking at wireshark now showed a more typical flow of data.
Now the connection would ensure flow control as you would expect and TCP segments were closer to what was
advertised as MSS and close to the MTU of the network.</p>