Tsinghua-Tencent 100K Tutorial

Hello, welcome to the tutorial of Tsinghua-Tencent 100k, in this tutorial, we will show you:

  • How to run our model with TT100K dataset
  • The structure of TT100K
  • How to observe TT100K
  • How to evaluate your result

How to run our model with TT100K dataset

First we need to download and extract the dataset and code from website http://cg.cs.tsinghua.edu.cn/traffic-sign/, Make sure your disk have 100G free space at least.

In [ ]:
mkdir TT100K && cd TT100K
wget http://cg.cs.tsinghua.edu.cn/traffic-sign/data_model_code/data.zip
wget http://cg.cs.tsinghua.edu.cn/traffic-sign/data_model_code/code.zip
unzip data.zip
unzip code.zip

Next, we have to build caffe code.

In [ ]:
cd code/script
./compile.sh

Then, we need to build the dataset into the LMDB format, and calculate the image mean for caffe training.

In [ ]:
mkdir -p ../../data/lmdb
./1_convert_lmdb.sh
./2_calc_img_mean.sh

Finally, we start the training process.

In [ ]:
../caffe/build/tools/caffe train --solver ../model/solver.prototxt --gpu 0

The training begin! The model converges in 40K iterations, it takes about 3 to 4 days.

The structure of TT100K

Explanation of directories:

code

  • caffe: caffe code that we used
  • model: our caffe model file
  • script: scripts used in this tutorial.
  • python: some python utils
    • anno_func.py: some useful utils function
    • data_show.ipynb: example of displaying images
    • deployer.ipynb: example of deployer
    • eval_check.ipynb: example of evaluating

data

  • train: all train pictures
  • test: all test pictures
  • other: some data that exclude from train and test
  • marks: standard traffic sign picture that are used for data agumentation
  • annotations.json: json file which contains the annotations of the pictures
  • results: results of our model and fast-rcnn

How to observe TT100K

In this section, we will show you the overview of this dataset, first we import the neccesery libraries.

In [8]:
cd ../python/

/mnt/hd1/from_linux/data/TT100K/code/python
In [9]:
import json
import pylab as pl
import random
import numpy as np
import cv2
import anno_func
%matplotlib inline

Then, we load the json annotation file and list of the test images's id

In [10]:
datadir = "../../data/"

filedir = datadir + "/annotations.json"
ids = open(datadir + "/test/ids.txt").read().splitlines()

annos = json.loads(open(filedir).read())

Let's take a look about annotation json file,

annos has two keys:

In [12]:
annos.keys()
Out[12]:
[u'imgs', u'types']

annos['types'] contains types we have in TT100K

In [14]:
",".join(annos['types'])
Out[14]:
u'i1,i10,i11,i12,i13,i14,i15,i2,i3,i4,i5,il100,il110,il50,il60,il70,il80,il90,io,ip,p1,p10,p11,p12,p13,p14,p15,p16,p17,p18,p19,p2,p20,p21,p22,p23,p24,p25,p26,p27,p28,p3,p4,p5,p6,p7,p8,p9,pa10,pa12,pa13,pa14,pa8,pb,pc,pg,ph1.5,ph2,ph2.1,ph2.2,ph2.4,ph2.5,ph2.8,ph2.9,ph3,ph3.2,ph3.5,ph3.8,ph4,ph4.2,ph4.3,ph4.5,ph4.8,ph5,ph5.3,ph5.5,pl10,pl100,pl110,pl120,pl15,pl20,pl25,pl30,pl35,pl40,pl5,pl50,pl60,pl65,pl70,pl80,pl90,pm10,pm13,pm15,pm1.5,pm2,pm20,pm25,pm30,pm35,pm40,pm46,pm5,pm50,pm55,pm8,pn,pne,po,pr10,pr100,pr20,pr30,pr40,pr45,pr50,pr60,pr70,pr80,ps,pw2,pw2.5,pw3,pw3.2,pw3.5,pw4,pw4.2,pw4.5,w1,w10,w12,w13,w16,w18,w20,w21,w22,w24,w28,w3,w30,w31,w32,w34,w35,w37,w38,w41,w42,w43,w44,w45,w46,w47,w48,w49,w5,w50,w55,w56,w57,w58,w59,w60,w62,w63,w66,w8,wo,i6,i7,i8,i9,ilx,p29,w29,w33,w36,w39,w4,w40,w51,w52,w53,w54,w6,w61,w64,w65,w67,w7,w9,pax,pd,pe,phx,plx,pmx,pnl,prx,pwx,w11,w14,w15,w17,w19,w2,w23,w25,w26,w27,pl0,pl4,pl3,pm2.5,ph4.4,pn40,ph3.3,ph2.6'

You can sample an id and visualize the image by using our utils.

In [29]:
imgid = random.sample(ids, 1)[0]
print imgid

imgdata = anno_func.load_img(annos, datadir, imgid)
imgdata_draw = anno_func.draw_all(annos, datadir, imgid, imgdata)
pl.figure(figsize=(20,20))
pl.imshow(imgdata_draw)

78307
((1106.3541259765625, 977.7344970703125), (18.043243408203125, 20.239809036254883), 122.83193969726562)
Out[29]:
<matplotlib.image.AxesImage at 0x7fbee59b8b90>

In sample 78307, we have 3 marks. We can dig a little bit more by print the json file.

In [30]:
annos['imgs'][imgid]
Out[30]:
{u'id': 78307,
 u'objects': [{u'bbox': {u'xmax': 1116.309,
    u'xmin': 1097.11,
    u'ymax': 987.6815,
    u'ymin': 968.116},
   u'category': u'p5',
   u'ellipse': [[1106.3541259765625, 977.7344970703125],
    [18.043243408203125, 20.239809036254883],
    122.83193969726562],
   u'ellipse_org': [[1111.86, 970.323],
    [1097.9, 974.499],
    [1101.05, 985.312],
    [1106.48, 987.428],
    [1116.38, 977.588]]},
  {u'bbox': {u'xmax': 1394.2022,
    u'xmin': 1318.11,
    u'ymax': 1000.3625,
    u'ymin': 931.884},
   u'category': u'w16',
   u'polygon': [[1358.12, 930.862], [1316.63, 997.996], [1395.79, 998.196]]},
  {u'bbox': {u'xmax': 2051.0886,
    u'xmin': 1995.29,
    u'ymax': 944.9281,
    u'ymin': 888.406},
   u'category': u'w15',
   u'polygon': [[2046.58, 925.997],
    [2024.55, 888.818],
    [1996.45, 944.184],
    [2044.19, 944.463],
    [2047.57, 944.46]]}],
 u'path': u'test/78307.jpg'}

The structure of the annotation are show below:

  • path: string, the relative path of the image
  • objects:[], all marks in this image
    • category: string, the category of the mark.
    • bbox: {xmin,ymin,xmax,ymax:float}, the bounding box of the mark
    • ellipse_org: [[x1,y1],[x2,y2],...], optional, the edge points of the mark
    • ellipse: [[x_o,y_o],[r_l,r_s],angle], optional, the ellipse mask of the mark, [x_o,y_o] is the center of the ellipse, [r_l,r_s] is the long and the short radis, angle is the rotation angle of the ellipse.
    • polygon: [[x1,y1],[x2,y2],...], optional, the edge points of the polygon mask

How to evaluate your result

First, load our results, you can replace it with your results.

Function eval_annos will tell you how many marks are right, wrong and missed,

some arguments can be used during evaluate, iou are the bounding box overlap ratio

In [31]:
result_anno_file = "./../results/ours_result_annos.json"
results_annos = json.loads(open(result_anno_file).read())
sm = anno_func.eval_annos(annos, results_annos, iou=0.5, types=anno_func.type45, check_type=True)
print sm['report']

iou:0.5, size:[0,400), types:[pl120, ...total 45...], accuracy:0.883742291038, recall:0.874334674802

We provide some utils to help you analysis your results,

For example, show the accuracy-recall curve on different size.

In [39]:
import sys
def get_acc_res(results_annos, **argv):
    scs = [ obj['score'] for k,img in results_annos['imgs'].items() for obj in img['objects']]
    scs = sorted(scs)
    accs = [0]
    recs = [1]
    for i, score in enumerate(np.linspace(0, scs[-1], 100)):
        sm = anno_func.eval_annos(annos, results_annos, iou=0.5, check_type=True, types=anno_func.type45, minscore=score, **argv)
        print "\r%s %s %s" % (i, score, sm['report']), 
        sys.stdout.flush()
        accs.append(sm['accuracy'])
        if len(accs)>=2 and accs[-1]<accs[-2]:
            accs[-1] = accs[-2]
        recs.append(sm['recall'])
    accs.append(1)
    recs.append(0)
    return accs, recs
sizes = [0,32,96,400]
ac_rc = []

for i in range(4):
    if i==3:
        l=sizes[0]
        r=sizes[-1]
    else:
        l=sizes[i]
        r=sizes[i+1]
    acc1, recs1 = get_acc_res(results_annos, minboxsize=l, maxboxsize=r)
    #acc2, recs2 = get_acc_res(results_annos2, minboxsize=l, maxboxsize=r)
    #ac_rc.append([acc1, recs1, acc2, recs2])
    ac_rc.append([acc1, recs1])
    
    pl.figure()
    pl.plot(acc1, recs1, label='ours')
    #pl.plot(acc2, recs2, label='fast-rcnn')
    pl.xlabel("accuracy")
    pl.ylabel("recall")
    pl.title("size: (%s,%s]"%(l,r))

    pl.legend(bbox_to_anchor=(0, 0), loc="lower left")
    #pl.savefig("/home/randon/data/newdata/results/ac-rc%s.pdf"%i)
#_ = pl.hist(scs, bins=100)

99 1285.0 iou:0.5, size:[0,400), types:[pl120, ...total 45...], accuracy:1.0, recall:0.000129819550824

Report on different size and types

In [37]:
test_annos = results_annos
minscore=40

sm = anno_func.eval_annos(annos, test_annos, iou=0.5, check_type=True, types=anno_func.type45,
                         minboxsize=0,maxboxsize=400,minscore=minscore)
print sm['report']
sm = anno_func.eval_annos(annos, test_annos, iou=0.5, check_type=True, types=anno_func.type45,
                         minboxsize=0,maxboxsize=32,minscore=minscore)
print sm['report']
sm = anno_func.eval_annos(annos, test_annos, iou=0.5, check_type=True, types=anno_func.type45,
                         minboxsize=32,maxboxsize=96,minscore=minscore)
print sm['report']
sm = anno_func.eval_annos(annos, test_annos, iou=0.5, check_type=True, types=anno_func.type45,
                         minboxsize=96,maxboxsize=400,minscore=minscore)
print sm['report']

for tp in anno_func.type45:
    sm = anno_func.eval_annos(annos, test_annos, iou=0.5, check_type=True, types=[tp],minscore=minscore)
    print sm['report']

iou:0.5, size:[0,400), types:[pl120, ...total 45...], accuracy:0.883742291038, recall:0.874334674802
iou:0.5, size:[0,32), types:[pl120, ...total 45...], accuracy:0.841992882562, recall:0.809165526676
iou:0.5, size:[32,96), types:[pl120, ...total 45...], accuracy:0.90686873625, recall:0.923344947735
iou:0.5, size:[96,400), types:[pl120, ...total 45...], accuracy:0.915277777778, recall:0.865965834428
iou:0.5, size:[0,400), types:i2, accuracy:0.75974025974, recall:0.823943661972
iou:0.5, size:[0,400), types:i4, accuracy:0.868312757202, recall:0.91341991342
iou:0.5, size:[0,400), types:i5, accuracy:0.95041322314, recall:0.912698412698
iou:0.5, size:[0,400), types:il100, accuracy:0.948717948718, recall:0.948717948718
iou:0.5, size:[0,400), types:il60, accuracy:0.975806451613, recall:0.864285714286
iou:0.5, size:[0,400), types:il80, accuracy:0.845454545455, recall:0.96875
iou:0.5, size:[0,400), types:io, accuracy:0.785234899329, recall:0.87969924812
iou:0.5, size:[0,400), types:ip, accuracy:0.885964912281, recall:0.7890625
iou:0.5, size:[0,400), types:p10, accuracy:0.761904761905, recall:0.919540229885
iou:0.5, size:[0,400), types:p11, accuracy:0.888198757764, recall:0.863179074447
iou:0.5, size:[0,400), types:p12, accuracy:0.921875, recall:0.893939393939
iou:0.5, size:[0,400), types:p19, accuracy:0.939393939394, recall:0.939393939394
iou:0.5, size:[0,400), types:p23, accuracy:0.912621359223, recall:0.912621359223
iou:0.5, size:[0,400), types:p26, accuracy:0.796428571429, recall:0.92531120332
iou:0.5, size:[0,400), types:p27, accuracy:0.88, recall:0.936170212766
iou:0.5, size:[0,400), types:p3, accuracy:0.742857142857, recall:0.896551724138
iou:0.5, size:[0,400), types:p5, accuracy:0.878787878788, recall:0.95867768595
iou:0.5, size:[0,400), types:p6, accuracy:0.744680851064, recall:0.897435897436
iou:0.5, size:[0,400), types:pg, accuracy:0.909090909091, recall:0.909090909091
iou:0.5, size:[0,400), types:ph4, accuracy:0.823529411765, recall:0.756756756757
iou:0.5, size:[0,400), types:ph4.5, accuracy:0.85, recall:0.85
iou:0.5, size:[0,400), types:ph5, accuracy:0.875, recall:0.717948717949
iou:0.5, size:[0,400), types:pl100, accuracy:0.905982905983, recall:0.981481481481
iou:0.5, size:[0,400), types:pl120, accuracy:0.987804878049, recall:0.931034482759
iou:0.5, size:[0,400), types:pl20, accuracy:0.9, recall:0.803571428571
iou:0.5, size:[0,400), types:pl30, accuracy:0.892682926829, recall:0.892682926829
iou:0.5, size:[0,400), types:pl40, accuracy:0.891304347826, recall:0.921348314607
iou:0.5, size:[0,400), types:pl5, accuracy:0.911458333333, recall:0.870646766169
iou:0.5, size:[0,400), types:pl50, accuracy:0.854395604396, recall:0.914705882353
iou:0.5, size:[0,400), types:pl60, accuracy:0.95652173913, recall:0.802919708029
iou:0.5, size:[0,400), types:pl70, accuracy:0.888888888889, recall:0.909090909091
iou:0.5, size:[0,400), types:pl80, accuracy:0.877133105802, recall:0.934545454545
iou:0.5, size:[0,400), types:pm20, accuracy:0.933333333333, recall:0.857142857143
iou:0.5, size:[0,400), types:pm30, accuracy:0.878787878788, recall:0.90625
iou:0.5, size:[0,400), types:pm55, accuracy:0.72, recall:0.947368421053
iou:0.5, size:[0,400), types:pn, accuracy:0.930810810811, recall:0.891304347826
iou:0.5, size:[0,400), types:pne, accuracy:0.931782945736, recall:0.910606060606
iou:0.5, size:[0,400), types:po, accuracy:0.785074626866, recall:0.676092544987
iou:0.5, size:[0,400), types:pr40, accuracy:0.921875, recall:0.936507936508
iou:0.5, size:[0,400), types:w13, accuracy:0.806451612903, recall:0.806451612903
iou:0.5, size:[0,400), types:w32, accuracy:0.95, recall:0.558823529412
iou:0.5, size:[0,400), types:w55, accuracy:0.970588235294, recall:0.55
iou:0.5, size:[0,400), types:w57, accuracy:0.907407407407, recall:0.803278688525
iou:0.5, size:[0,400), types:w59, accuracy:0.816326530612, recall:0.666666666667
iou:0.5, size:[0,400), types:wo, accuracy:0.458333333333, recall:0.289473684211

And plot the mark you are wrong or missed

In [43]:
while True:
    imgid = random.sample(results_annos['imgs'].keys(), 1)[0]
    if len(sm['wrong']["imgs"][imgid]["objects"]) and len(sm['miss']["imgs"][imgid]["objects"]):
        break
print imgid

pl.figure(figsize=(20,20))
imgdata = anno_func.load_img(annos, datadir, imgid)
imgdata = anno_func.draw_all(sm['right'], datadir, imgid, imgdata, (0,1,0), False)
imgdata = anno_func.draw_all(sm['wrong'], datadir, imgid, imgdata, (1,0,0), False)
imgdata = anno_func.draw_all(sm['miss'], datadir, imgid, imgdata, (0,0,1), False)
pl.imshow(imgdata)

7807
Out[43]:
<matplotlib.image.AxesImage at 0x7fbebd652550>
In [ ]: