Python + OpenSCAD

I’ve been interested in automating the process of designing building structures according to a set of input parameters, and having a software application determine all of the pieces necessary to meet structural engineering principals and applicable building codes.

An ideal tool for this is OpenSCAD, which is a 3D modeling program that uses code as input.

There is a nice Python library which is a wrapper around the OpenSCAD input language: https://github.com/SolidCode/SolidPython

Here is a proof of concept. It’s working well so far!

[ERROR] InnoDB: Cannot add field in table because after adding it, the row size is 8572 which is greater than maximum allowed size (8126) for a record on index leaf page.

One day, suddenly, we couldn’t reload a production backup.  Oh no!  It took a bit of digging around but here is the issue, should you find yourself running into it:

Using the InnoDB COMPACT ROW_FORMAT, there are some rules InnoDB uses to decide how much data to put on the first page and how much to put on overflow pages.  While VARCHAR and TEXT type columns are typically stored on overflow pages with no problem, when using the COMPACT row format, up to the first 767 bytes of each field will be stored on the primary page.

This error happens when the amount of actual data exceeds the allows limit for a row size on the main page, which is about 50% of the innodb_page_size variable.

https://mariadb.com/kb/en/library/innodb-compact-row-format/

If you have a circumstance with say, 11 TEXT or VARCHAR fields, each with at least 767 bytes of data, and running an innodb_page_size of 16k (which results in 8126 being max row size on main page) …… then 767*11=8437 stored on main page is greater than 8126 which is the “maximum allowed size for a record on index leaf page”, and you will get this error.

The solution is to investigate the use of either DYNAMIC or COMPRESSED row format, depending on your application.

 

Python Mutable vs Immutable

Here is a basic introduction to immutable and immutable types in python.

In python there are two types of data… mutable and immutable. Numbers, strings, boolean, tuples, and other simple types are immutable. Dicts, lists, sets, objects, classes, and other complex types are mutable.

When you say:

a = [1,2,3]
b = a

You’ve created a single mutable list in memory, assigned a to point to it, and then assigned b to point to it. It’s the same thing in memory.

Therefore when you mutate it (modify it):

b[0] = 3

It is a modification (mutation) of the index [0] of the value which b points to at that same memory location.

However, when you replace it:

b = [0,0,0]

It is creating a new mutable list in memory and assigning b to point at it.


Check out the id() function. It will tell you the “address” of any variable. You can see which names are pointing to the same memory location with id(varname).


Bonus: Every value in python is passed by reference… meaning that when you assign it to a variable it simply causes that variable to point to that value where it was in memory. Having immutable types allows python to “reuse” the same memory location for common immutable types.

Consider some common values when the interpreter starts up.  You can see here there are a lot of variables pointing at the memory location held by abc.  cpython, at least, is smart enough to realize that the value `abc` is already stored in memory and because it is immutable, just returns that same memory address.

>>> import sys
>>> sys.getrefcount('abc')
68
>>> sys.getrefcount(100)
110
>>> sys.getrefcount(2)
6471

However, a value that is definitely not present would return 2. This has to do with the fact that a couple of references to that value were in-use during the call to sys.getrefcount

>>> sys.getrefcount('nope not me.  I am definitely not here already.')
2

Notice that an empty tuple has a lot of references:

>>> sys.getrefcount(tuple())
34571

But an empty list has no extra references:

>>> sys.getrefcount(list())
1

Why is this? Because tuple is immutable so it is fine to share that value across any number of variables. However, lists are mutable so they MUST NOT be shared across arbitrary variables or changes to one would affect the others.

Incidentally, this is also why you must NEVER use mutable types as default argument values to functions. Consider this innocent little function:

>>> def foo(value=[]):
...     value.append(1)
...     print(value)
...
...

When you call it you might expect to get [1] printed…

>>> foo()
[1]

However, when you call it again, you prob. won’t expect to get [1,1] out… ???

>>> foo()
[1, 1]

And on and on…

>>> foo()
[1, 1, 1]

>>> foo()
[1, 1, 1, 1]

WHY IS THIS? Because default arguments to functions are evaluated once during function definition, and not at function run time. That way if you use a mutable value as a default argument value, then you will be stuck with that one value, mutating in unexpected ways as the function is called multiple times.

The proper way to do it is this:

>>> def foo(value=None):
...     if value is None:
...         value = []
...     value.append(1)
...     print(value)
...
...
>>>
>>> foo()
[1]
>>> foo()
[1]
>>> foo()
[1]

Example of Python Generator Function.

One of the great features of Python is Generator Functions.  Generator functions allow you to convert any function to a generator function by simply including the yield keyword somewhere in the function body.  When a generator function is called, the response is a generator object, which can be iterated over among other things.

To boil this down to basics, generator functions allow you the programmer to create a function which yields one value at a time (and pauses until the next value is requested) until you decide it is done.  This opens endless possibilities for converting sequences, creating sequences, filtering, and more.

The following example is a generator function which will take a simple list or iterator and return pairs of (element, previous_element).  This is a great use for generators and the yield statement.

Generator Function:

def lineandlast(listish, first=True, last=True):
    iterator = iter(listish)
    lastline = next(iterator)

    if first:
      yield lastline, None

    for line in iterator:
        yield line, lastline
        lastline = line

    if last:
        yield None, lastline

Example:

for line, last in lineandlast([1,2,3,4,5]):
    print(line, last)

Output:

1 None
2 1
3 2
4 3
5 4
None 5

Notes:
There are two keyword arguments, first and last, which can be used to control the output of the first and last items on the output example above.

VisPy Sinewave Demo

VisPy is quite an interesting tool.  They say it is for “scientific visualization”.  Here is an example of that.  I took a demo from their github page and added a little code to generate a sinewave.  The cool thing is that the graph, scaling, panning, zooming, and redrawing all come out of the box.

I previously wrote about installing VisPy.  Thanks to the great efforts of some unnamed people, Python on Windows is really working nicely now.

Here is a screenshot:

vis

Here is the code:


# -*- coding: utf-8 -*-
# Copyright (c) Vispy Development Team. All Rights Reserved.
# Distributed under the (new) BSD License. See LICENSE.txt for more info.
"""
Demonstration of InfiniteLine visual.
"""
import sys
import numpy as np
from vispy import app, scene
# vertex positions of data to draw
N = 200
pos = np.zeros((N, 2), dtype=np.float32)
x_lim = [50., 1750.]
y_lim = [-2., 2.]
pos[:, 0] = np.linspace(x_lim[0], x_lim[1], N)
pos[:, 1] = np.random.normal(size=N)
pos1 = np.zeros((20000,2), dtype=np.float32)
color1 = np.ones((20000,4), dtype=np.float32)
pos1[0,0] = 0.0
pos1[0,1] = 15.0
pos1[1,0] = 100.0
pos1[1,1] = -15.0
import math
for x in range(20000):
pos1[x,0] = x*10
pos1[x,1] = math.sin(x/20.) * 40.
# color array
color = np.ones((N, 4), dtype=np.float32)
color[:, 0] = np.linspace(0, 1, N)
color[:, 1] = color[::-1, 0]
canvas = scene.SceneCanvas(keys='interactive', show=True)
grid = canvas.central_widget.add_grid(spacing=0)
viewbox = grid.add_view(row=0, col=1, camera='panzoom')
# add some axes
x_axis = scene.AxisWidget(orientation='bottom')
x_axis.stretch = (1, 0.1)
grid.add_widget(x_axis, row=1, col=1)
x_axis.link_view(viewbox)
y_axis = scene.AxisWidget(orientation='left')
y_axis.stretch = (0.1, 1)
grid.add_widget(y_axis, row=0, col=0)
y_axis.link_view(viewbox)
# add a line plot inside the viewbox
#line = scene.Line(pos, color, parent=viewbox.scene)
line1 = scene.Line(pos1, color1, parent=viewbox.scene)
# add vertical lines
vert_line1 = scene.InfiniteLine(100, [1.0, 0.0, 0.0, 1.0],
parent=viewbox.scene)
vert_line2 = scene.InfiniteLine(549.2, [0.0, 1.0, 0.0, 1.0], vertical=True,
parent=viewbox.scene)
# add horizontal lines
hor_line1 = scene.InfiniteLine(0.3, [1.0, 0.0, 1.0, 1.0], vertical=False,
parent=viewbox.scene)
hor_line2 = scene.InfiniteLine(-5.1, [1.0, 1.0, 0.0, 1.0], vertical=False,
parent=viewbox.scene)
# auto-scale to see the whole line.
viewbox.camera.set_range((0,1000), (-100,100))
if __name__ == '__main__' and sys.flags.interactive == 0:
app.run()

view raw

vis.py

hosted with ❤ by GitHub

Centos AMI on Amazon EC2 – How To Resize The Disk

When using the official CentOS Linux AMI on EC2, it comes (currently) with only an 8GB partition.  Want to resize that?  Here is how…

First make sure you start a small (t2.micro or similar) instance to do this work with.  It should be based on the Amazon Linux AMI.  You should have root access to it.  Let’s call it “fix-the-disk-instance”.

  1. Stop the NEW instance
    1. Go to EC2 > Instances
    2. Select the Instance
    3. Make sure the Instance has a name that makes sense (e.g. snap.appcove.net)
    4. Click “Actions” > “Instance State” > Stop
  2. Detach the block device
    1. Go to EC2 > Volumes
    2. Find the volume by looking under Attachment Information for the instance name (e.g. snap.appcove.net)
    3. Select the volume
    4. Click “Actions” > “Detach Volume”
  3. Attach the block device to “fix-the-disk-instance”
    1. Go to EC2 > Volumes
    2. Find the volume by looking under Attachment Information for the instance name (e.g. snap.appcove.net)
    3. Select the volume
    4. Click “Actions” > “Attach Volume”
    5. Attach it to “fix-the-disk-instance” so we can fix it up
  4. Start “fix-the-disk-instance” and login
    1. Go to EC2 > Volumes
    2. Select the Instance “fix-the-disk-instance”
    3. Click “Actions” > “Instance State” > Start
    4. Wait for it to start
    5. SSH into it and escalate to root
  5. Run `parted` command and print list
    1. As root, run the following command: parted
    2. Type the following command: print all<enter>
    3. Find your disk in the output and note the path (e.g. /dev/xvdf)
    4. Quit the program by typing: quit<enter>
  6. Resize the partition:
    1. Run fdisk on the device path found above (e.g. fdisk /dev/xvdf)
    2. List the partitions
      1. Type: p<enter>
      2. Take note of the “Start” column (e.g. 2048)
    3. Delete the partition
      1. Type: d<enter>
    4. Create a new one by typing:
      1. n (for new)
      2. p (for primary)
      3. 1 (for first partition)
      4. Type first sector the same as the Start column above (e.g. 2048)
      5. Last sector should be the default (end of disk)
      6. w (for writing to disk)
  7. Now check the partition and resize it
    1. Run this command making sure to use the right device path (but add a 1 to the end for the first partition)
      1. e2fsck -f /dev/xvdf1
    2. Resize the partition making sure to use the right device path (but add a 1 to the end for the first partition)
      1. resize2fs /dev/xvdf1
  8. Shutdown the instance “fix-the-disk-instance”
    1. Go to EC2 > Instances
    2. Select the Instance “fix-the-disk-instance”
    3. Click “Actions” > “Instance State” > Stop
  9. Detach the block device and reattach it to the original server
    1. Go to EC2 > Volumes
    2. Find the volume by looking under Attachment Information for the instance name (e.g. fix-the-device-instance).
    3. Make sure it is the right volume by looking at the ID/Size.
    4. Select the volume
    5. Click “Actions” > “Detach Volume”
    6. Click “Actions” > “Attach Volume”
    7. IMPORTANT: enter “/dev/sda1” as the device so it attaches as the root volume
    8. Attach it to the original server
  10. Start “fix-the-disk-instance” and login
    1. Go to EC2 > Volumes
    2. Select the original instance (e.g. “snap.appcove.net”)
    3. Click “Actions” > “Instance State” > Start
    4. Wait for it to start

 

And that’s all!

 

 

2016-03-20 - B162957

2016-03-20 - B164242

2016-03-20 - B164338

 

 

nginx: how to specify a default server

Several years ago when I started using nginx, I was under the mistaken assumption that

server_name _;

was a wildcard server name and would be used if no other server names matched.

Nope.

I made a change on a production system, adding a new site on an existing IP address.  What harm could that cause, right?

After several clients quickly and graciously notified us that the wrong site was coming up when you visited their domain, I quickly tracked the problem down.

First you need to realize that server_name _ is actually not special.  It is just a non-match.

Second you need to realize that in the event of no matches, nginx will select the first server{} block and use that.

This means that the ORDER of your server blocks is critical if you are using `server_name _;`. 

In our case, the order was incorrect, and my new domain was picking up all requests for that IP address.  I tell this because I believe a number of system administrators have this incorrectly configured and waiting to bite them.

There is a better way.

The nginx `listen` directive includes a `default_server` option that looks like this:

server{
   listen 1.2.3.4:80 default_server;
   ...
}

From http://wiki.nginx.org/HttpCoreModule#listen

If the directive has the default_server parameter, then the enclosing server {…} block will be the default server for the address:port pair. This is useful for name-based virtual hosting where you wish to specify the default server block for hostnames that do not match any server_name directives. If there are no directives with the default_server parameter, then the default server will be the first server block in which the address:port pair appears.

The moral of the story

It is better to use the correct mechanism (above) than relying on a single non-matching server_name.

I hope someone finds this useful!

Reference: http://stackoverflow.com/questions/9454764/nginx-server-name-wildcard-or-catch-all

 

 

 

Arduino Bipolar Stepper Circuit

I got a cool little 4-wire bipolar stepper motor and wanted to drive it via Arduino.

I based the design on this reference: 

http://arduino.cc/en/Reference/StepperBipolarCircuit (credit for following image belongs there as well)

 

 

bipolar_stepper_four_pins

Here is how it looks for real.  Kind of a mess of wires, but you know what?  It works great.

IMG_2031

 

 

 

Introducing FileStruct (for Python)

FileStruct is a lightweight and fast file-cache / file-server designed for web-applications.  It solves the problems of “where do I save all of those uploads” that has been encountered time and time again.  FileStruct uses the local filesystem, but in a sensible way (keeping permissions sane), and with the ability to secure it to a reasonable level.

https://github.com/appcove/FileStruct/

Here is a simple example of taking an image upload, resizing, and saving it:

with client.TempDir() as TempDir:
   open(TempDir.FilePath('upload.jpg'), 'wb').write(mydata)
   TempDir.ResizeImage('upload.jpg', 'resize.jpg', '100x100')
   hash1 = TempDir.Save('upload.jpg')
   hash2 = TempDir.Save('resize.jpg')

Design Goals

Immutable Files

FileStruct is designed to work with files represented by the SHA-1 hash of their contents. This means that all files in FileStruct are immutable.

High Performance

FileStruct is designed as a local repository of file data accessable (read/write) by an application or web application. All operations are local I/O operations and therefore, very fast.

Where possible, streaming hash functions are used to prevent iterating over a file twice.

Direct serving from Nginx

FileStruct is designed so that Nginx can serve files directly from it’s Data directory using an X-Accel-Redirect header. For more information on this Nginx configuration directive, see http://wiki.nginx.org/XSendfile

Assuming that nginx runs under nginx user and file database is owned by the fileserver group, nginx needs to be in thefileserver group to serve files:

# usermod -a -G fileserver nginx

Secure

FileStruct is designed to be as secure as your hosting configuration. Where possible, a dedicated user should be allocated to read/write to FileStruct, and the database directory restricted to this user.

Simple

FileStruct is designed to be incredibly simple to use.

File Manipulaion

FileStruct is designed to simplify common operations on files, especially uploaded files. Image resizing for thumbnails is supported.

Temporary File Management

FileStruct is designed to simplify the use of Temp Files in an application. The API supports creation of a temporary directory, placing files in it, Ingesting files into FileStruct, and deleting the directory when completed (or retaining it in the event of an error)

Garbage Collection

FileStruct is designed to retain files until garbage collection is performed. Garbage collection consists of telling FileStruct what files you are interested in keeping, and having it move the remaining files to the trash.

Backup and Sync with Rsync

FileStruct is designed to work seamlessly with rsync for backups and restores.

Atomic operations

At the point a file is inserted or removed from FileStruct, it is a filesystem move operation. This means that under no circumstances will a file exist in FileStruct that has contents that do not match the name of the file.

No MetaData

FileStruct is not designed to store MetaData. It is designed to store file content. There may be several “files” which refer to the same content. empty.logempty.txt, and empty.ini may all refer to the empty fileData/da/39/da39a3ee5e6b4b0d3255bfef95601890afd80709. However, this file will be retained as long as any aspect of the application still uses it.

Automatic De-Duplication

Because file content is stored in files with the hash of the content, automatic file-level de-duplication occurs. When a file is pushed to FileStruct that already exists, there is no need to write it again.

This carries the distinct benifit of being able to use the same FileStruct database across multiple projects if desired, because the content of file Data/da/39/da39a3ee5e6b4b0d3255bfef95601890afd80709 is always the same, regardless of the application that placed it there.

Note: In the event that multiple instances or applications use the same database, the garbage collection routine MUST take all references to a given hash into account, across all applications that use the database. Otherwise, it would be easy to delete data that should be retained.