1
00:00:00,539 --> 00:00:03,539
foreign

2
00:00:08,240 --> 00:00:12,480
this is the last presentation of this

3
00:00:10,860 --> 00:00:13,440
block and then we have like some lunch

4
00:00:12,480 --> 00:00:16,260
break

5
00:00:13,440 --> 00:00:18,720
so I have here another add light part

6
00:00:16,260 --> 00:00:20,760
time hit up member so all of our homes

7
00:00:18,720 --> 00:00:24,300
we just seen yourself to a developer

8
00:00:20,760 --> 00:00:26,580
with over a 15 years of experiencing uh

9
00:00:24,300 --> 00:00:28,439
worked in lots of GIS projects

10
00:00:26,580 --> 00:00:31,260
it's gonna give a nice presentation

11
00:00:28,439 --> 00:00:35,340
about flask or full stack application in

12
00:00:31,260 --> 00:00:37,559
using tiles very interesting uh okay

13
00:00:35,340 --> 00:00:42,079
Owen it's with you

14
00:00:37,559 --> 00:00:42,079
30 minutes two thank you

15
00:00:44,760 --> 00:00:48,840
uh thank you so much for joining me

16
00:00:47,640 --> 00:00:51,180
today

17
00:00:48,840 --> 00:00:55,199
um I'm really was really happy to see

18
00:00:51,180 --> 00:00:58,820
such a big geospatial Presence at the

19
00:00:55,199 --> 00:01:02,219
conference so that's that's exciting

20
00:00:58,820 --> 00:01:03,899
so as as Pedro was saying

21
00:01:02,219 --> 00:01:05,400
um a little bit about me I'm currently a

22
00:01:03,899 --> 00:01:07,860
senior software engineer working for

23
00:01:05,400 --> 00:01:11,100
anditi which is an organization that

24
00:01:07,860 --> 00:01:13,020
provides geospatial visualization and

25
00:01:11,100 --> 00:01:15,000
Analysis I've been a software engineer

26
00:01:13,020 --> 00:01:17,280
for over 15 years and I've worked on a

27
00:01:15,000 --> 00:01:19,920
variety of projects but I'm passionate

28
00:01:17,280 --> 00:01:22,020
about geospatial data management and

29
00:01:19,920 --> 00:01:23,820
delivery

30
00:01:22,020 --> 00:01:28,080
uh so

31
00:01:23,820 --> 00:01:30,380
the goal of uh this this presentation is

32
00:01:28,080 --> 00:01:33,180
to talk about transforming and

33
00:01:30,380 --> 00:01:37,520
optimizing large image data sets for

34
00:01:33,180 --> 00:01:39,720
dynamic use on a front-end interface so

35
00:01:37,520 --> 00:01:41,400
when I

36
00:01:39,720 --> 00:01:43,140
well I've been working I've worked for

37
00:01:41,400 --> 00:01:46,079
organizations in the past where we've we

38
00:01:43,140 --> 00:01:48,900
get large image data sets often from

39
00:01:46,079 --> 00:01:51,000
aerial photography

40
00:01:48,900 --> 00:01:53,460
um and what you can get it from you know

41
00:01:51,000 --> 00:01:55,680
aerial photography or satellite data and

42
00:01:53,460 --> 00:01:59,939
you get the files that you get from that

43
00:01:55,680 --> 00:02:02,159
are typically single large data sets and

44
00:01:59,939 --> 00:02:05,360
you know you need to be able to put them

45
00:02:02,159 --> 00:02:05,360
in a format that you could use

46
00:02:05,520 --> 00:02:08,819
so

47
00:02:06,840 --> 00:02:11,400
what we do is we take those large image

48
00:02:08,819 --> 00:02:12,860
data sets and we transform them into a

49
00:02:11,400 --> 00:02:15,480
slippy tile format

50
00:02:12,860 --> 00:02:17,819
and then we we store that data in a

51
00:02:15,480 --> 00:02:21,239
Cloud store and then we create an API

52
00:02:17,819 --> 00:02:25,200
endpoint to access that data and then we

53
00:02:21,239 --> 00:02:28,560
can also enhance the data through that

54
00:02:25,200 --> 00:02:30,980
API endpoint and manipulate that data in

55
00:02:28,560 --> 00:02:30,980
transition

56
00:02:32,940 --> 00:02:38,459
so you know why do we do this so when we

57
00:02:36,900 --> 00:02:40,020
get those large files sometimes those

58
00:02:38,459 --> 00:02:43,920
large files can be you know multiple

59
00:02:40,020 --> 00:02:45,599
gigabytes in size and it's pretty

60
00:02:43,920 --> 00:02:49,140
difficult to download all that data to

61
00:02:45,599 --> 00:02:52,140
your computer straight away so

62
00:02:49,140 --> 00:02:54,000
uh we we put it into this tiled format

63
00:02:52,140 --> 00:02:57,060
and this tiled format allows us to more

64
00:02:54,000 --> 00:02:59,459
easily stream that data to the end user

65
00:02:57,060 --> 00:03:02,340
um you would be probably familiar with

66
00:02:59,459 --> 00:03:04,800
this using similar Technologies like

67
00:03:02,340 --> 00:03:07,620
Google Maps which really

68
00:03:04,800 --> 00:03:09,120
um certainly created a big

69
00:03:07,620 --> 00:03:11,580
um you know industry in terms of

70
00:03:09,120 --> 00:03:14,459
providing you know rusted data for for

71
00:03:11,580 --> 00:03:16,620
General use and they were one of the

72
00:03:14,459 --> 00:03:18,180
original inventors of a slippy map tile

73
00:03:16,620 --> 00:03:18,840
format

74
00:03:18,180 --> 00:03:21,959
um

75
00:03:18,840 --> 00:03:26,459
and uh yeah it allows us to more easily

76
00:03:21,959 --> 00:03:29,000
access that data on a web app or a

77
00:03:26,459 --> 00:03:29,000
mobile device

78
00:03:30,800 --> 00:03:37,340
for this example I'm using a data set

79
00:03:35,459 --> 00:03:40,500
that I obtained from open aerial map

80
00:03:37,340 --> 00:03:42,900
open aerial map is a great open source

81
00:03:40,500 --> 00:03:44,700
repository of Aero imagery where you can

82
00:03:42,900 --> 00:03:47,519
you can download aerial imagery that

83
00:03:44,700 --> 00:03:49,620
other people put up online aerial map is

84
00:03:47,519 --> 00:03:52,379
a great example because it provides you

85
00:03:49,620 --> 00:03:55,440
know raw files raw Geo tests that are

86
00:03:52,379 --> 00:03:58,080
pre-built that you can you can download

87
00:03:55,440 --> 00:03:59,340
and you can test so if you if you wanted

88
00:03:58,080 --> 00:04:00,900
to try out some of this sort of stuff

89
00:03:59,340 --> 00:04:02,519
yourself or you know play around with

90
00:04:00,900 --> 00:04:06,000
some geospatial tools it just gives you

91
00:04:02,519 --> 00:04:08,400
the ability to to have some open imagery

92
00:04:06,000 --> 00:04:12,180
to be able to use for this particular

93
00:04:08,400 --> 00:04:13,860
example I'm using a town in Laos

94
00:04:12,180 --> 00:04:14,760
uh so

95
00:04:13,860 --> 00:04:17,220
um

96
00:04:14,760 --> 00:04:18,479
a bit more about the slippy tile map

97
00:04:17,220 --> 00:04:21,900
format so

98
00:04:18,479 --> 00:04:24,600
the slippy tile map format Works uh in a

99
00:04:21,900 --> 00:04:27,479
pyramid structure where you have at the

100
00:04:24,600 --> 00:04:30,180
top you have one tile that would

101
00:04:27,479 --> 00:04:32,460
represent a a large area so one tire

102
00:04:30,180 --> 00:04:34,259
would represent the entire Earth

103
00:04:32,460 --> 00:04:38,520
and then at the bottom where you have

104
00:04:34,259 --> 00:04:40,440
finer detail with tiles that represent

105
00:04:38,520 --> 00:04:42,300
close to the ground so at the top you

106
00:04:40,440 --> 00:04:43,979
could have a single tile all the way

107
00:04:42,300 --> 00:04:47,220
down to billions of tiles it could

108
00:04:43,979 --> 00:04:51,259
represent the higher detail at ground

109
00:04:47,220 --> 00:04:55,860
level these tiles are separated up into

110
00:04:51,259 --> 00:04:58,580
rows columns and zoom levels so nxy and

111
00:04:55,860 --> 00:04:58,580
Z coordinate

112
00:05:01,520 --> 00:05:06,419
so for for this example and there is a

113
00:05:04,680 --> 00:05:09,800
lot of geospatial libraries that you can

114
00:05:06,419 --> 00:05:13,199
you can use for manipulating roster data

115
00:05:09,800 --> 00:05:15,600
one particular tool that is uh is is

116
00:05:13,199 --> 00:05:18,540
great to use as well is called rust.io

117
00:05:15,600 --> 00:05:22,259
but for this example I'm actually using

118
00:05:18,540 --> 00:05:24,720
a tool called gdal the reason why I use

119
00:05:22,259 --> 00:05:26,520
gdal is because gdo is one of the

120
00:05:24,720 --> 00:05:29,160
underlying pinning Technologies of

121
00:05:26,520 --> 00:05:30,660
things like rust or IO

122
00:05:29,160 --> 00:05:32,940
um and uh

123
00:05:30,660 --> 00:05:35,400
so you you kind of have a bit more

124
00:05:32,940 --> 00:05:37,740
access to the functionality

125
00:05:35,400 --> 00:05:39,600
um so it gives me um in some cases it's

126
00:05:37,740 --> 00:05:41,580
given me a bit more freedom to to work

127
00:05:39,600 --> 00:05:43,440
with the data but for a lot of General

128
00:05:41,580 --> 00:05:44,340
cases things like raster are really good

129
00:05:43,440 --> 00:05:46,919
as well

130
00:05:44,340 --> 00:05:49,639
uh then I'm going to store the starter

131
00:05:46,919 --> 00:05:53,060
in AWS

132
00:05:49,639 --> 00:05:57,360
using the S3

133
00:05:53,060 --> 00:06:00,560
and I'm also using Botto 3 which is part

134
00:05:57,360 --> 00:06:03,020
of the AWS SDK for python

135
00:06:00,560 --> 00:06:07,199
is a C plus plus

136
00:06:03,020 --> 00:06:10,280
project and we use Python bindings to

137
00:06:07,199 --> 00:06:10,280
leverage that technology

138
00:06:11,100 --> 00:06:20,039
so this is uh an example of how I'd use

139
00:06:14,580 --> 00:06:22,380
gdal warp and what it takes in is the

140
00:06:20,039 --> 00:06:24,419
tile bound so

141
00:06:22,380 --> 00:06:28,380
um we'll do we do a calculation to

142
00:06:24,419 --> 00:06:30,120
figure out what bounds uh the the tail

143
00:06:28,380 --> 00:06:32,280
the tail represents in this particular

144
00:06:30,120 --> 00:06:35,340
example looking at balance in 4326

145
00:06:32,280 --> 00:06:37,680
because there are different projections

146
00:06:35,340 --> 00:06:40,979
um I guess the other aspect that I could

147
00:06:37,680 --> 00:06:43,020
go into as well is that um this uh this

148
00:06:40,979 --> 00:06:44,940
method also allows us to use data that's

149
00:06:43,020 --> 00:06:47,460
produced in different Native projections

150
00:06:44,940 --> 00:06:48,900
which is something that you know

151
00:06:47,460 --> 00:06:50,699
different systems use and produce

152
00:06:48,900 --> 00:06:54,360
especially at like different countries

153
00:06:50,699 --> 00:06:56,759
and regions and we can we can use this

154
00:06:54,360 --> 00:06:59,460
to essentially create like one format

155
00:06:56,759 --> 00:07:01,800
that allows all geospatial software to

156
00:06:59,460 --> 00:07:03,840
use so

157
00:07:01,800 --> 00:07:05,000
this uh in this example we've got those

158
00:07:03,840 --> 00:07:10,139
bounds

159
00:07:05,000 --> 00:07:13,259
produces the tiles in 256 by 256 pixels

160
00:07:10,139 --> 00:07:14,759
which is the standard size of a tile in

161
00:07:13,259 --> 00:07:17,280
the slippy map format although you can

162
00:07:14,759 --> 00:07:20,180
use five twelve by 512 for high

163
00:07:17,280 --> 00:07:20,180
definition tiles

164
00:07:21,380 --> 00:07:25,860
and then we use bottle three to upload

165
00:07:24,180 --> 00:07:27,199
the tiles so this is an example this is

166
00:07:25,860 --> 00:07:30,919
an example of

167
00:07:27,199 --> 00:07:33,960
uploading a a tile to

168
00:07:30,919 --> 00:07:37,979
uploading an image and uploading entail

169
00:07:33,960 --> 00:07:40,440
to an F3 S3 store although you could use

170
00:07:37,979 --> 00:07:41,720
this this type of Curry to essentially

171
00:07:40,440 --> 00:07:44,819
upload

172
00:07:41,720 --> 00:07:47,940
any type of file to

173
00:07:44,819 --> 00:07:50,720
uh an S3 store as well

174
00:07:47,940 --> 00:07:50,720
oh sorry

175
00:07:51,240 --> 00:07:54,680
let me break out of this for a second

176
00:07:58,039 --> 00:08:02,639
so I'm just going to run a script now

177
00:08:00,539 --> 00:08:05,039
and so this script

178
00:08:02,639 --> 00:08:07,199
I just wanted to show you kind of just

179
00:08:05,039 --> 00:08:09,900
to give you more of an active example

180
00:08:07,199 --> 00:08:12,180
of how you I'm running this script and

181
00:08:09,900 --> 00:08:13,979
this is uploading the tiles so you could

182
00:08:12,180 --> 00:08:16,680
see this as really as like two different

183
00:08:13,979 --> 00:08:19,440
processes so it's the tile pipeline of

184
00:08:16,680 --> 00:08:22,199
like creating that data and then the the

185
00:08:19,440 --> 00:08:24,360
visualization of creating the API to

186
00:08:22,199 --> 00:08:27,900
deliver that data

187
00:08:24,360 --> 00:08:31,319
and so I can I'll run this script

188
00:08:27,900 --> 00:08:34,440
now you can see that as the uh the

189
00:08:31,319 --> 00:08:34,440
script runs

190
00:08:36,240 --> 00:08:41,039
um

191
00:08:38,360 --> 00:08:43,320
it's gone through and it's created Tiles

192
00:08:41,039 --> 00:08:45,720
at different Zoom levels different Tiles

193
00:08:43,320 --> 00:08:47,279
at different Zoom levels and the output

194
00:08:45,720 --> 00:08:50,839
is

195
00:08:47,279 --> 00:08:50,839
in this map directory

196
00:08:53,399 --> 00:08:58,640
so now you can see that

197
00:08:55,860 --> 00:09:02,160
uh what it's taken is one big large file

198
00:08:58,640 --> 00:09:06,560
and then it's separated up into

199
00:09:02,160 --> 00:09:06,560
these individual map tiles

200
00:09:15,200 --> 00:09:23,700
so uh The Next Step we're going to look

201
00:09:19,260 --> 00:09:24,839
at is creating an API to modify the data

202
00:09:23,700 --> 00:09:27,779
in stream

203
00:09:24,839 --> 00:09:30,540
so what do I mean by that so what the

204
00:09:27,779 --> 00:09:32,160
what I'm what I want to be able to do is

205
00:09:30,540 --> 00:09:34,500
I want to be able to modify this data

206
00:09:32,160 --> 00:09:37,380
and change properties of the data but

207
00:09:34,500 --> 00:09:39,839
without saving it to a server so being

208
00:09:37,380 --> 00:09:43,019
able to to modify that data in the byte

209
00:09:39,839 --> 00:09:45,180
stream and be able to give that those

210
00:09:43,019 --> 00:09:47,120
those that real-time data update to the

211
00:09:45,180 --> 00:09:51,360
end user

212
00:09:47,120 --> 00:09:53,779
so we in our our system retrieves a tile

213
00:09:51,360 --> 00:09:59,720
from that data store

214
00:09:53,779 --> 00:09:59,720
then it's our API then uses pillow

215
00:09:59,880 --> 00:10:05,100
to modify that data and then that

216
00:10:01,860 --> 00:10:08,180
modified data gets then gets returned to

217
00:10:05,100 --> 00:10:08,180
the client

218
00:10:08,700 --> 00:10:13,519
so I'm just going to give you an example

219
00:10:11,040 --> 00:10:13,519
of this

220
00:10:14,519 --> 00:10:20,519
so this this tool that I'm using is a

221
00:10:16,800 --> 00:10:21,620
qjs and qjs is a tool that you can use

222
00:10:20,519 --> 00:10:24,360
to

223
00:10:21,620 --> 00:10:28,459
visualize map tiles

224
00:10:24,360 --> 00:10:28,459
it's also a great open source project

225
00:10:29,580 --> 00:10:32,880
so what I'm going to load first is this

226
00:10:31,260 --> 00:10:34,140
is actually the original file and that's

227
00:10:32,880 --> 00:10:36,180
one of the reasons why you can see those

228
00:10:34,140 --> 00:10:37,200
bands associated with the red green and

229
00:10:36,180 --> 00:10:39,540
blue band

230
00:10:37,200 --> 00:10:41,880
and

231
00:10:39,540 --> 00:10:44,459
this is yeah so this is the the original

232
00:10:41,880 --> 00:10:47,779
original data and it was a single file

233
00:10:44,459 --> 00:10:47,779
so I can zoom into that

234
00:10:59,180 --> 00:11:04,200
so now I'm going to come over here and

235
00:11:02,040 --> 00:11:08,480
I'm going to start

236
00:11:04,200 --> 00:11:08,480
my my flask API endpoint

237
00:11:10,260 --> 00:11:13,460
[Applause]

238
00:11:18,120 --> 00:11:24,240
and so what it's done here is it's

239
00:11:20,459 --> 00:11:26,519
retrieving the data from that store

240
00:11:24,240 --> 00:11:28,740
and

241
00:11:26,519 --> 00:11:30,240
and by no means am I giving you an

242
00:11:28,740 --> 00:11:32,820
example of great usable data in this

243
00:11:30,240 --> 00:11:35,820
presentation but this is an example of

244
00:11:32,820 --> 00:11:38,100
how I've taken that that original image

245
00:11:35,820 --> 00:11:40,220
from the data store and then I've

246
00:11:38,100 --> 00:11:43,260
brightened it up substantially

247
00:11:40,220 --> 00:11:44,700
but in no but this this data never

248
00:11:43,260 --> 00:11:45,899
actually had to be like saved to the

249
00:11:44,700 --> 00:11:47,880
device and then modified and then

250
00:11:45,899 --> 00:11:50,120
returned it was all done in that flight

251
00:11:47,880 --> 00:11:50,120
stream

252
00:11:52,320 --> 00:11:59,459
this is an example of doing just a a PNG

253
00:11:56,640 --> 00:12:02,040
overlay so I had a single PNG that was

254
00:11:59,459 --> 00:12:05,399
saved on the server and then as it came

255
00:12:02,040 --> 00:12:07,140
through the API it pasted the PNG over

256
00:12:05,399 --> 00:12:08,519
the top of the image that it retrieved

257
00:12:07,140 --> 00:12:10,740
from that data store

258
00:12:08,519 --> 00:12:13,680
this is something you could also I've

259
00:12:10,740 --> 00:12:16,560
just put a a simple PNG here but you

260
00:12:13,680 --> 00:12:19,260
could use you know polygons or Point

261
00:12:16,560 --> 00:12:21,420
Vector data and paste that over the top

262
00:12:19,260 --> 00:12:25,040
or any other assets that you would like

263
00:12:21,420 --> 00:12:25,040
to show on on your tiles

264
00:12:27,600 --> 00:12:33,740
and then lastly I was going to show you

265
00:12:30,600 --> 00:12:36,480
an example of applying a histogram now

266
00:12:33,740 --> 00:12:39,300
again you know I know it's actually

267
00:12:36,480 --> 00:12:42,540
doing the histogram to individual Tiles

268
00:12:39,300 --> 00:12:45,300
at the moment so it's essentially

269
00:12:42,540 --> 00:12:48,300
so you can see it's a bit patchy in

270
00:12:45,300 --> 00:12:51,980
terms of How It's displayed but I think

271
00:12:48,300 --> 00:12:51,980
what I really wanted to demonstrate

272
00:12:52,380 --> 00:12:57,540
by doing this is just the simple

273
00:12:54,600 --> 00:12:59,220
flexibility that you have so there's

274
00:12:57,540 --> 00:13:01,760
really nothing that you can't do with

275
00:12:59,220 --> 00:13:01,760
this data

276
00:13:13,620 --> 00:13:19,320
um so this is this is a very simple

277
00:13:16,139 --> 00:13:21,540
example of the API endpoint that we just

278
00:13:19,320 --> 00:13:23,519
used but I restricted it just to show

279
00:13:21,540 --> 00:13:25,620
you the um

280
00:13:23,519 --> 00:13:28,399
the way that actually brightens the data

281
00:13:25,620 --> 00:13:28,399
and

282
00:13:28,680 --> 00:13:31,560
just as I thought it would take a long

283
00:13:30,300 --> 00:13:35,220
time to go through all the other

284
00:13:31,560 --> 00:13:38,160
functions but see you can see here that

285
00:13:35,220 --> 00:13:40,980
we're retrieving those parameters so the

286
00:13:38,160 --> 00:13:43,680
layer name the zoom the row the column

287
00:13:40,980 --> 00:13:46,800
so we can locate that tile

288
00:13:43,680 --> 00:13:50,639
then we after we get that that inventory

289
00:13:46,800 --> 00:13:52,740
back from the S3 service we're applying

290
00:13:50,639 --> 00:13:55,220
that brightness filter directly to that

291
00:13:52,740 --> 00:13:57,839
data enhancing that data and then

292
00:13:55,220 --> 00:14:00,320
returning it directly back in the in the

293
00:13:57,839 --> 00:14:00,320
byte stream

294
00:14:07,139 --> 00:14:09,560
um so

295
00:14:07,920 --> 00:14:12,300
[Applause]

296
00:14:09,560 --> 00:14:13,440
uh after all of that what I was gonna I

297
00:14:12,300 --> 00:14:15,779
wanna what I'd really like to show you

298
00:14:13,440 --> 00:14:18,120
as well it's just an example of um you

299
00:14:15,779 --> 00:14:18,899
know one of the use cases that

300
00:14:18,120 --> 00:14:21,300
um

301
00:14:18,899 --> 00:14:24,540
you can use it use this data for so

302
00:14:21,300 --> 00:14:27,600
after you've created that API

303
00:14:24,540 --> 00:14:28,920
you can use that for

304
00:14:27,600 --> 00:14:31,500
um you know full stack development

305
00:14:28,920 --> 00:14:33,959
creating you know mapping applications

306
00:14:31,500 --> 00:14:37,160
where you could use custom raster data

307
00:14:33,959 --> 00:14:39,720
that could be retrieved and used by that

308
00:14:37,160 --> 00:14:44,120
that data source

309
00:14:39,720 --> 00:14:48,180
in this example we're using a ccmjs

310
00:14:44,120 --> 00:14:51,720
which utilizes the 3js library the

311
00:14:48,180 --> 00:14:55,139
JavaScript 3.js library to create a 3D

312
00:14:51,720 --> 00:14:57,620
globe on that 3D Globe then we can place

313
00:14:55,139 --> 00:15:02,060
that raster imagery

314
00:14:57,620 --> 00:15:06,180
this project was also created with react

315
00:15:02,060 --> 00:15:08,639
and I distributed the

316
00:15:06,180 --> 00:15:12,779
the front end

317
00:15:08,639 --> 00:15:17,240
onto an S3 bucket and the data is being

318
00:15:12,779 --> 00:15:17,240
delivered by a Lambda function on AWS

319
00:15:19,199 --> 00:15:22,199
foreign

320
00:15:39,860 --> 00:15:44,459
is provided by Bing Maps so that would

321
00:15:42,600 --> 00:15:46,320
be satellite imagery produced in the

322
00:15:44,459 --> 00:15:48,620
same using the same processors that we

323
00:15:46,320 --> 00:15:48,620
just saw

324
00:15:50,180 --> 00:15:54,560
so we're just going to zoom into the map

325
00:15:57,839 --> 00:16:03,720
okay so and here we've got that that map

326
00:16:01,620 --> 00:16:06,839
that we were looking at before but now

327
00:16:03,720 --> 00:16:08,880
we're seeing that all of the uh the

328
00:16:06,839 --> 00:16:12,180
tiles

329
00:16:08,880 --> 00:16:14,720
on this 3D globe and we can navigate

330
00:16:12,180 --> 00:16:14,720
around

331
00:16:15,660 --> 00:16:23,459
and just an example to show you

332
00:16:19,800 --> 00:16:25,320
so we're using that bing bing data

333
00:16:23,459 --> 00:16:27,480
service and that's the the quality of

334
00:16:25,320 --> 00:16:29,820
the data that they're producing

335
00:16:27,480 --> 00:16:32,160
uh from from satellite data and this is

336
00:16:29,820 --> 00:16:34,680
the aerial imagery that was produced by

337
00:16:32,160 --> 00:16:37,079
that data set you can see how

338
00:16:34,680 --> 00:16:39,920
much you can zoom in and see that that

339
00:16:37,079 --> 00:16:39,920
level of quality

340
00:16:40,320 --> 00:16:45,060
which I think is really quite good for

341
00:16:43,199 --> 00:16:48,180
um you know open source imagery there is

342
00:16:45,060 --> 00:16:50,160
a a few little hiccups in the data that

343
00:16:48,180 --> 00:16:51,720
I've seen But for something that was

344
00:16:50,160 --> 00:16:54,380
completely open source and available I

345
00:16:51,720 --> 00:16:54,380
thought it was really good

346
00:16:56,339 --> 00:17:01,620
and

347
00:16:58,920 --> 00:17:04,279
as part of this you can light it up and

348
00:17:01,620 --> 00:17:04,279
you can see the

349
00:17:06,079 --> 00:17:09,740
looking the wrong way

350
00:17:15,419 --> 00:17:19,919
yeah so you can that way you can combine

351
00:17:17,280 --> 00:17:21,839
different data sources which I wasn't

352
00:17:19,919 --> 00:17:23,579
going to go into today but you can see

353
00:17:21,839 --> 00:17:25,079
there's a there's a Terrain data source

354
00:17:23,579 --> 00:17:27,559
in the background then you've got that

355
00:17:25,079 --> 00:17:31,740
you can see that 2D

356
00:17:27,559 --> 00:17:34,919
imagery and see what is around those

357
00:17:31,740 --> 00:17:37,880
areas and you can also you know

358
00:17:34,919 --> 00:17:42,179
potentially like drape the uh the 2D

359
00:17:37,880 --> 00:17:44,580
imagery over the 3D terrain and other

360
00:17:42,179 --> 00:17:45,900
you know interesting way there is a lot

361
00:17:44,580 --> 00:17:48,260
of other interesting ways to use the

362
00:17:45,900 --> 00:17:48,260
data

363
00:17:55,380 --> 00:18:00,059
so I feel like I might have gone quite

364
00:17:57,179 --> 00:18:02,220
fast today but that was uh that was my

365
00:18:00,059 --> 00:18:04,500
full presentation

366
00:18:02,220 --> 00:18:05,650
um so uh thank you so much for uh for

367
00:18:04,500 --> 00:18:13,440
joining me today

368
00:18:05,650 --> 00:18:15,539
[Applause]

369
00:18:13,440 --> 00:18:18,179
because we finished earlier we can take

370
00:18:15,539 --> 00:18:21,380
questions so any questions from the

371
00:18:18,179 --> 00:18:21,380
audience and working

372
00:18:23,580 --> 00:18:26,600
so I have one here

373
00:18:27,299 --> 00:18:33,120
I'm just curious if you've looked at

374
00:18:29,340 --> 00:18:36,059
dynamically generating uh slippy tiles

375
00:18:33,120 --> 00:18:38,880
themselves from like an upstream data

376
00:18:36,059 --> 00:18:41,780
source so that you can avoid having to

377
00:18:38,880 --> 00:18:43,860
process all of that original data source

378
00:18:41,780 --> 00:18:46,200
have you looked at doing that sort of

379
00:18:43,860 --> 00:18:48,539
thing dynamically well sorry what was

380
00:18:46,200 --> 00:18:50,220
that like so so

381
00:18:48,539 --> 00:18:51,480
if you're pilling in some data like that

382
00:18:50,220 --> 00:18:54,120
and you've got and then you slice that

383
00:18:51,480 --> 00:18:56,340
up into tiles that's cool but what if

384
00:18:54,120 --> 00:18:57,960
the original data set is absolutely huge

385
00:18:56,340 --> 00:19:00,059
and you only want to bring in

386
00:18:57,960 --> 00:19:01,980
subsets of that dynamically can you

387
00:19:00,059 --> 00:19:04,799
generate the slippy tiles

388
00:19:01,980 --> 00:19:06,660
dynamically on demand uh yeah there is

389
00:19:04,799 --> 00:19:07,980
ways of doing that but I've actually

390
00:19:06,660 --> 00:19:10,860
found like especially with the large

391
00:19:07,980 --> 00:19:14,520
data sets it's it's more difficult

392
00:19:10,860 --> 00:19:18,120
um I've had success with creating larger

393
00:19:14,520 --> 00:19:21,240
image data stores on single servers and

394
00:19:18,120 --> 00:19:23,220
I can use a gdo and actually I think it

395
00:19:21,240 --> 00:19:25,500
you if you were looking for a you know

396
00:19:23,220 --> 00:19:26,820
like an ad hoc solution I can do I can

397
00:19:25,500 --> 00:19:28,919
you can potentially set up something

398
00:19:26,820 --> 00:19:31,679
that can cut and deliver those tiles on

399
00:19:28,919 --> 00:19:33,059
on one one server but in terms of

400
00:19:31,679 --> 00:19:37,440
actually being able to provide that

401
00:19:33,059 --> 00:19:40,200
service to a more distributed user base

402
00:19:37,440 --> 00:19:43,500
then I found that that it's um

403
00:19:40,200 --> 00:19:46,919
yeah it's just not not efficient enough

404
00:19:43,500 --> 00:19:48,240
so um but um but yeah if if you were

405
00:19:46,919 --> 00:19:51,179
just doing this for say for instance

406
00:19:48,240 --> 00:19:53,280
like an internal Network there is ways

407
00:19:51,179 --> 00:19:54,660
of just just doing it more like creating

408
00:19:53,280 --> 00:19:58,220
the data on the Fly and being able to

409
00:19:54,660 --> 00:19:58,220
keep that data in its original format

410
00:20:02,160 --> 00:20:05,059
any other question

411
00:20:06,539 --> 00:20:10,220
trying to find hands

412
00:20:13,400 --> 00:20:20,760
are up there I have to

413
00:20:17,940 --> 00:20:23,460
you go there cool

414
00:20:20,760 --> 00:20:27,860
so let's check on the

415
00:20:23,460 --> 00:20:27,860
Discord channel in the meanwhile yeah

416
00:20:34,679 --> 00:20:39,900
yeah I was gonna ask about the um when

417
00:20:37,799 --> 00:20:42,179
you overlay the logo as like a watermark

418
00:20:39,900 --> 00:20:43,799
if you could do geospecific and I think

419
00:20:42,179 --> 00:20:46,919
you've kind of answered this with your

420
00:20:43,799 --> 00:20:48,179
last slide here so if you

421
00:20:46,919 --> 00:20:50,160
uh

422
00:20:48,179 --> 00:20:52,260
yeah could you explain how you did the

423
00:20:50,160 --> 00:20:55,080
thank you because that's kind of like I

424
00:20:52,260 --> 00:20:56,640
assume annotating a single tile based on

425
00:20:55,080 --> 00:20:58,799
some conditional parameters or something

426
00:20:56,640 --> 00:21:02,460
like that oh sure actually the the thank

427
00:20:58,799 --> 00:21:04,559
you was part of the

428
00:21:02,460 --> 00:21:07,260
the the framework that I provided it

429
00:21:04,559 --> 00:21:10,700
wasn't wasn't the tiles but um the but

430
00:21:07,260 --> 00:21:13,380
you can do geo spatial specific data so

431
00:21:10,700 --> 00:21:15,200
you could overlay that with any sort of

432
00:21:13,380 --> 00:21:17,220
data that you want you know Vector data

433
00:21:15,200 --> 00:21:19,380
potentially other raster data if you

434
00:21:17,220 --> 00:21:22,440
wanted to layer like additional rusted

435
00:21:19,380 --> 00:21:24,299
so you know the possibilities are uh

436
00:21:22,440 --> 00:21:26,340
endless this uh

437
00:21:24,299 --> 00:21:28,380
this was a part of the uh the cesium

438
00:21:26,340 --> 00:21:30,780
framework so if I was to actually like

439
00:21:28,380 --> 00:21:33,360
go on the side

440
00:21:30,780 --> 00:21:35,700
okay I think I could you could do that

441
00:21:33,360 --> 00:21:37,440
if you did it further Upstream would

442
00:21:35,700 --> 00:21:39,600
that be in the

443
00:21:37,440 --> 00:21:41,280
generating the rest of I guess it would

444
00:21:39,600 --> 00:21:43,559
be would be or did you do that with the

445
00:21:41,280 --> 00:21:46,020
flask where you're kind of on the Fly

446
00:21:43,559 --> 00:21:47,880
manipulating it oh no so this is this is

447
00:21:46,020 --> 00:21:49,740
uh this is all part of the front end

448
00:21:47,880 --> 00:21:52,500
framework the um the CCM front-end

449
00:21:49,740 --> 00:21:55,200
framework but if you wanted to splice in

450
00:21:52,500 --> 00:21:56,940
other Geo type data would is that

451
00:21:55,200 --> 00:21:57,720
something you could do with the tiles so

452
00:21:56,940 --> 00:21:59,760
I think you've got to mentioned

453
00:21:57,720 --> 00:22:02,659
annotating it with some other bits yeah

454
00:21:59,760 --> 00:22:02,659
no absolutely

455
00:22:02,700 --> 00:22:06,539
and so would that be it'd be done

456
00:22:04,919 --> 00:22:07,799
through the flash type piece you had

457
00:22:06,539 --> 00:22:08,820
where you're doing the brightness and

458
00:22:07,799 --> 00:22:10,860
other bits because that was kind of

459
00:22:08,820 --> 00:22:13,500
applied to all tiles

460
00:22:10,860 --> 00:22:15,240
uh yeah so I mean as they've said the

461
00:22:13,500 --> 00:22:16,740
kind of the possibility is endless so I

462
00:22:15,240 --> 00:22:18,900
mean the the data that's coming through

463
00:22:16,740 --> 00:22:20,340
that the python API is like you're

464
00:22:18,900 --> 00:22:21,720
retrieving essentially like single flat

465
00:22:20,340 --> 00:22:23,760
tiles yeah

466
00:22:21,720 --> 00:22:26,640
um but in terms of the freedom that you

467
00:22:23,760 --> 00:22:29,280
have like using like utilizing pillow or

468
00:22:26,640 --> 00:22:30,059
other other libraries I think with the

469
00:22:29,280 --> 00:22:32,820
um

470
00:22:30,059 --> 00:22:34,500
uh uh you know but I didn't go into it

471
00:22:32,820 --> 00:22:36,659
entirely but when I was doing that

472
00:22:34,500 --> 00:22:39,000
histogram data was actually using a CV

473
00:22:36,659 --> 00:22:41,340
to do the histogram so I was actually

474
00:22:39,000 --> 00:22:42,659
second gate naming navigating around

475
00:22:41,340 --> 00:22:44,100
um pillow

476
00:22:42,659 --> 00:22:45,299
um but there is like endless

477
00:22:44,100 --> 00:22:46,980
possibilities of like how you can

478
00:22:45,299 --> 00:22:49,020
manipulate that data and how you can

479
00:22:46,980 --> 00:22:49,740
apply you know geospatial assets and

480
00:22:49,020 --> 00:22:51,659
then

481
00:22:49,740 --> 00:22:52,919
um you know retrieve it to that so you

482
00:22:51,659 --> 00:22:54,059
could do it at that layer because pillow

483
00:22:52,919 --> 00:22:56,100
is just sort of general image

484
00:22:54,059 --> 00:22:57,659
manipulation so you could do the Geo

485
00:22:56,100 --> 00:23:00,419
stuff at that

486
00:22:57,659 --> 00:23:02,640
quite a bit as well okay yeah I mean so

487
00:23:00,419 --> 00:23:05,520
I guess it and it really comes down to

488
00:23:02,640 --> 00:23:06,960
like efficiency so if you wanted to

489
00:23:05,520 --> 00:23:09,539
um if you wanted to do that manipulate

490
00:23:06,960 --> 00:23:11,640
manipulation there you could another

491
00:23:09,539 --> 00:23:13,320
service that you can use with uh you

492
00:23:11,640 --> 00:23:14,520
know geospatial data one of the more

493
00:23:13,320 --> 00:23:18,299
common

494
00:23:14,520 --> 00:23:19,980
um data sources is uh um like MBT Vector

495
00:23:18,299 --> 00:23:21,360
data tiles yeah

496
00:23:19,980 --> 00:23:22,740
um one of the great things about using

497
00:23:21,360 --> 00:23:24,480
like you know Vector data so actually

498
00:23:22,740 --> 00:23:26,340
having to be able to split up those two

499
00:23:24,480 --> 00:23:28,620
Services is

500
00:23:26,340 --> 00:23:30,480
um that you can have the vector data

501
00:23:28,620 --> 00:23:33,059
embedded within that tile and then being

502
00:23:30,480 --> 00:23:35,039
able to you know directly query that

503
00:23:33,059 --> 00:23:36,179
data in the front-end interface which is

504
00:23:35,039 --> 00:23:39,419
something you can't do with the roster

505
00:23:36,179 --> 00:23:40,860
data so generally with the products that

506
00:23:39,419 --> 00:23:41,539
I've made in the past

507
00:23:40,860 --> 00:23:45,480
um

508
00:23:41,539 --> 00:23:48,000
I typically do split that split those

509
00:23:45,480 --> 00:23:50,700
data sources up yeah and and you could

510
00:23:48,000 --> 00:23:52,200
do that in the previous desktop app you

511
00:23:50,700 --> 00:23:55,679
had but you'd also do that with cesium

512
00:23:52,200 --> 00:23:57,179
and have those two different layers

513
00:23:55,679 --> 00:23:58,799
so could you explain that just a little

514
00:23:57,179 --> 00:23:59,760
bit yeah sorry it's really weird okay oh

515
00:23:58,799 --> 00:24:01,200
there we go

516
00:23:59,760 --> 00:24:02,700
um so you could do that both in this

517
00:24:01,200 --> 00:24:06,360
front end as well so you kind of had the

518
00:24:02,700 --> 00:24:09,000
image data set and then have the vectors

519
00:24:06,360 --> 00:24:10,260
as a separate layer yeah so that and

520
00:24:09,000 --> 00:24:12,419
that would that would be typically done

521
00:24:10,260 --> 00:24:14,760
in like the front end framework that we

522
00:24:12,419 --> 00:24:20,220
use um you know I said for this one I'm

523
00:24:14,760 --> 00:24:22,200
using CCM uh J CCM JS which is

524
00:24:20,220 --> 00:24:23,640
um you know allows you to to be able to

525
00:24:22,200 --> 00:24:27,480
display

526
00:24:23,640 --> 00:24:29,880
um you know 3D data 2D data um

527
00:24:27,480 --> 00:24:32,880
but you know we uh I've I've also used

528
00:24:29,880 --> 00:24:34,740
like open layers and mapbox and other

529
00:24:32,880 --> 00:24:36,360
different front-end Frameworks where the

530
00:24:34,740 --> 00:24:38,760
the processes would be slightly

531
00:24:36,360 --> 00:24:40,500
different one of the things that I

532
00:24:38,760 --> 00:24:42,720
really do love about CCM is it's

533
00:24:40,500 --> 00:24:44,640
actually from my own experience it's the

534
00:24:42,720 --> 00:24:47,460
most difficult to use but it has the

535
00:24:44,640 --> 00:24:50,360
most freedom in what you can do with it

536
00:24:47,460 --> 00:24:50,360
cool thank you

537
00:24:54,419 --> 00:24:57,320
any more questions

538
00:24:57,480 --> 00:25:03,320
oh I've got one and then you okay cool

539
00:25:00,720 --> 00:25:03,320
cheers

540
00:25:05,580 --> 00:25:09,720
um so kind of tacking off the question

541
00:25:08,520 --> 00:25:11,460
um about

542
00:25:09,720 --> 00:25:13,080
the vector data and the rest of the data

543
00:25:11,460 --> 00:25:14,760
is it possible to apply a similar

544
00:25:13,080 --> 00:25:16,919
technique to the

545
00:25:14,760 --> 00:25:19,919
um the actual Terrain map itself and

546
00:25:16,919 --> 00:25:22,039
sort of patch in height data into the

547
00:25:19,919 --> 00:25:22,039
map

548
00:25:22,380 --> 00:25:28,320
um well with um I think if if I was to

549
00:25:26,820 --> 00:25:30,240
so yeah there's definitely different

550
00:25:28,320 --> 00:25:33,179
data data sources so you can have

551
00:25:30,240 --> 00:25:34,260
terrain data you can have 3D

552
00:25:33,179 --> 00:25:36,840
um

553
00:25:34,260 --> 00:25:37,919
you know 3D tiles which are you know

554
00:25:36,840 --> 00:25:38,940
essentially you can create like 3D

555
00:25:37,919 --> 00:25:41,580
buildings

556
00:25:38,940 --> 00:25:45,120
um and at the moment I work on a project

557
00:25:41,580 --> 00:25:47,760
where we we bring in lidar data or like

558
00:25:45,120 --> 00:25:50,279
you know Point data so and typically we

559
00:25:47,760 --> 00:25:53,340
combine those data sources like on on

560
00:25:50,279 --> 00:25:56,460
that front-end interface if

561
00:25:53,340 --> 00:25:58,860
um you can manipulate the terrain

562
00:25:56,460 --> 00:26:01,080
actually with the front-end framework so

563
00:25:58,860 --> 00:26:02,940
um the the front-end framework can

564
00:26:01,080 --> 00:26:04,320
manipulate that data as it comes in and

565
00:26:02,940 --> 00:26:06,720
say like for instance if you wanted to

566
00:26:04,320 --> 00:26:08,700
say in this particular area I don't want

567
00:26:06,720 --> 00:26:10,640
to display terrain you could just ask

568
00:26:08,700 --> 00:26:12,600
the front-end framework to remove that

569
00:26:10,640 --> 00:26:14,159
and then if you want to bring in another

570
00:26:12,600 --> 00:26:16,080
data source I'd recommend doing that

571
00:26:14,159 --> 00:26:19,220
more on the front end than the back end

572
00:26:16,080 --> 00:26:19,220
awesome thanks

573
00:26:20,100 --> 00:26:28,039
um you mentioned oh sorry yeah

574
00:26:24,480 --> 00:26:33,779
this is oh okay yeah sorry you mentioned

575
00:26:28,039 --> 00:26:36,900
that you applied that histogram to one

576
00:26:33,779 --> 00:26:38,760
of the pictures right one of the small

577
00:26:36,900 --> 00:26:42,120
tiles

578
00:26:38,760 --> 00:26:45,860
and you got those patchy results

579
00:26:42,120 --> 00:26:49,039
can you apply it on a much larger scale

580
00:26:45,860 --> 00:26:52,559
as well and then

581
00:26:49,039 --> 00:26:54,779
break it up afterwards or how would you

582
00:26:52,559 --> 00:26:57,000
do that yeah no it's a great question

583
00:26:54,779 --> 00:26:59,520
actually it's really what you saw was

584
00:26:57,000 --> 00:27:03,000
kind of um more or less up where I was

585
00:26:59,520 --> 00:27:05,279
up to in my own personal research so

586
00:27:03,000 --> 00:27:06,960
um what when I was you know

587
00:27:05,279 --> 00:27:08,880
experimenting with what uh what

588
00:27:06,960 --> 00:27:11,039
capabilities that I had I was able to

589
00:27:08,880 --> 00:27:12,740
create those histograms with those

590
00:27:11,039 --> 00:27:15,120
individual Tales

591
00:27:12,740 --> 00:27:17,640
I think that you know that's something

592
00:27:15,120 --> 00:27:19,980
that you possibly could do and actually

593
00:27:17,640 --> 00:27:22,740
be able to you know provide that

594
00:27:19,980 --> 00:27:24,120
histogram to large areas but that's

595
00:27:22,740 --> 00:27:26,100
something that I haven't experimented

596
00:27:24,120 --> 00:27:28,620
with yet but something I'm interested in

597
00:27:26,100 --> 00:27:31,340
experimenting with in the future

598
00:27:28,620 --> 00:27:31,340
thank you

599
00:27:31,500 --> 00:27:36,960
sorry if there's questions off topic um

600
00:27:34,020 --> 00:27:40,020
my team wants to self-host open street

601
00:27:36,960 --> 00:27:41,100
map tiles to be consumed by a leaflet

602
00:27:40,020 --> 00:27:43,260
client

603
00:27:41,100 --> 00:27:45,000
where should we begin

604
00:27:43,260 --> 00:27:48,299
so you want to use

605
00:27:45,000 --> 00:27:50,400
um open street map tiles and you want to

606
00:27:48,299 --> 00:27:52,500
use leaflet to be able to deliver them

607
00:27:50,400 --> 00:27:55,020
that's correct yeah

608
00:27:52,500 --> 00:27:56,640
um I mean I think if I was to tackle

609
00:27:55,020 --> 00:27:59,460
that project myself it would really just

610
00:27:56,640 --> 00:28:01,860
be like you know setting up a

611
00:27:59,460 --> 00:28:04,919
um you know a leaf lift project and then

612
00:28:01,860 --> 00:28:06,900
using the open street map tiles in that

613
00:28:04,919 --> 00:28:10,140
example that I showed you before that

614
00:28:06,900 --> 00:28:13,020
was uh you know open street maps

615
00:28:10,140 --> 00:28:16,799
uh here on the bottom so you know

616
00:28:13,020 --> 00:28:19,140
openstreetmaps has a open readily

617
00:28:16,799 --> 00:28:20,600
available API to be able to attain that

618
00:28:19,140 --> 00:28:24,480
data

619
00:28:20,600 --> 00:28:26,460
which is also rusted source which would

620
00:28:24,480 --> 00:28:28,080
have which can be you know constructed

621
00:28:26,460 --> 00:28:30,539
in a similar sort of fashion but

622
00:28:28,080 --> 00:28:32,039
essentially if you were just looking to

623
00:28:30,539 --> 00:28:33,299
get this data you know it's really just

624
00:28:32,039 --> 00:28:37,320
about going through that process of

625
00:28:33,299 --> 00:28:38,760
setting up the the leaflet system

626
00:28:37,320 --> 00:28:40,740
um you know personally I think that if I

627
00:28:38,760 --> 00:28:42,659
was going to do a project like that I

628
00:28:40,740 --> 00:28:44,220
would be I'd be looking at something

629
00:28:42,659 --> 00:28:46,919
like

630
00:28:44,220 --> 00:28:50,039
um open Street uh like open layers just

631
00:28:46,919 --> 00:28:52,500
to provide a bit more you know freedom

632
00:28:50,039 --> 00:28:54,120
in you know what how I could manipulate

633
00:28:52,500 --> 00:28:55,740
and display the data but you know at

634
00:28:54,120 --> 00:28:57,240
least that's still a great solution as

635
00:28:55,740 --> 00:28:59,520
well

636
00:28:57,240 --> 00:29:01,440
um but I think those those two

637
00:28:59,520 --> 00:29:03,539
technologies match up perfectly well so

638
00:29:01,440 --> 00:29:05,880
it's really just

639
00:29:03,539 --> 00:29:08,120
um I think there's any real challenges

640
00:29:05,880 --> 00:29:08,120
there

641
00:29:10,740 --> 00:29:16,500
all right so with that uh we ran out of

642
00:29:13,260 --> 00:29:18,360
time now but uh the Discord channel is

643
00:29:16,500 --> 00:29:19,799
always open for questions and if you can

644
00:29:18,360 --> 00:29:22,620
didn't pop up to check if there's any

645
00:29:19,799 --> 00:29:25,440
other questions about the talk so we're

646
00:29:22,620 --> 00:29:27,919
gonna now thank you ollie for this great

647
00:29:25,440 --> 00:29:27,919
presentation

648
00:29:31,380 --> 00:29:38,240
and here's our note in a mug for you as

649
00:29:34,200 --> 00:29:38,240
a thank you all right so