Paper-StripMaker-Sketch-Consolidation

Keywords: ClusterStrokes, Random Forest, Vector Sketch Consolidation

StripMaker: Perception-driven Learned Vector Sketch Consolidation

CHENXI LIU, University of British Columbia, Canada
TOSHIKI AOKI, University of Tokyo, Japan
MIKHAIL BESSMELTSEV, Université de Montréal, Canada
ALLA SHEFFER, University of British Columbia, Canada

https://www.cs.ubc.ca/labs/imager/tr/2023/stripmaker/

overview

WorkFlow

Inspiration

The Main Process

StripMakerWorkflow

Sketch-Consolidation-Process

RandomForest-Classifier

Innovation

  • avoid the need for an unsustainably large manually annotated learning corpus by leveraging observations about artist workflow and perceptual cues viewers employ when mentally consolidating sketches.

  • Our method is the first to use a principled classification-based approach to vector sketch consolidation.

Limitation

Our performance is constrained by data scarcity which prevents greater reliance on context, due to overfitting concerns.

More

Random Forest

👉More about Random Forest in PatternRecognition>>

StrongRelatedPaper

Dave Pagurek van Mossel, Chenxi Liu, Nicholas Vining, Mikhail Bessmeltsev, and Alla Sheffer. 2021. StrokeStrip: Joint Parameterization and Fitting of Stroke Clusters. ACM Trans. Graph. 40, 4 (July 2021), 50:1–50:18

Peng Xu, Timothy M Hospedales, Qiyue Yin, Yi-Zhe Song, Tao Xiang, and Liang Wang. 2022. Deep learning for free-hand sketch: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence (2022)

Jerry Yin, Chenxi Liu, Rebecca Lin, Nicholas Vining, Helge Rhodin, and Alla Sheffer. 2022. Detecting Viewer-Perceived Intended Vector Sketch Connectivity. ACM Transactions on Graphics 41 (2022). Issue 4.

Code

Since the author doesn’t upload codes, so I just guess the rough code framwork as follows:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
//.h
//used to get the likelyhood between different strips, so as to merge or split
class RandomForestClassfier
{
//StrokeStrip parameterization[van Mossel et al. 2021].
public:
RandomForestClassfier();
virtual ~RandomForestClassfier();
//stroke and stroke
virtual float getLikelihoodSS(Stroke& a, Stroke& b);
//stroke and strip
virtual float getLikelihoodSP(Stroke& a, vector<Stroke>& b);
//strip and strip
virtual float getLikelihoodPP(vector<Stroke>& a, vector<Stroke>& b);
};

class LocalClassifier : public RandomForestClassfier
{
public:
LocalClassifier();
~LocalClassifier();
//stroke and stroke
virtual float getLikelihoodSS(Stroke& a, Stroke& b);
//stroke and strip
virtual float getLikelihoodSP(Stroke& a, vector<Stroke>& b);
//strip and strip
virtual float getLikelihoodPP(vector<Stroke>& a, vector<Stroke>& b);
};
class GlobalClassifier : public RandomForestClassfier
{
public:
//stroke and stroke
virtual float getLikelihoodSS(Stroke& a, Stroke& b);
//stroke and strip
virtual float getLikelihoodSP(Stroke& a, vector<Stroke>& b);
//strip and strip
virtual float getLikelihoodPP(vector<Stroke>& a, vector<Stroke>& b);
};
class Stroke
{
private:
vector<Point> controlPoints; //each stroke is denoted by controlpoints, sorted by time
//Time t; //the time this stroke is drawed by user
vector<Point> samplePoints;
int stripIndx; //which strip this stroke belongs to
public:
bool getSamplePoints(float sampleRate);
bool setStripIndx(int idx);
int getStripIndx();
};
class VectorSketch
{
private:
vector<Stroke> Strokes; //the vector Sketch consists of raw strokes
vector<vector<Stroke>> Strips; //group the strokes to different strips
int strokeWidth;
public:
bool Sample();
bool LocalConsolidation();
bool GlobalConsolidation();

bool IsCompatible(vector<Stroke>& a, vector<Stroke>& b);
bool Merge(vector<Stroke>& a, vector<Stroke>& b);

bool IsFormjectionByYin2022(vector<Stroke>& a, vector<Stroke>& b);
void FittingCurvesByVanMossel2021();
};
//main.cpp
int main()
{
//Input, get the information of strokes
VectorSketch vectorSketch;

//Preprocess
vectorSketch.Sample();


//Step1. temporal consolidation
vectorSketch.LocalConsolidation();

//Step2. refinement
vectorSketch.GlobalConsolidation();

//Compare Results Experiments..
}

Sample, LocalConsolidation and GlobalConsolidation funtions can be implemented as follows:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
//Stroke.cpp
bool Stroke::getSamplePoints(float sampleRate)
{
for (float t = 0; t < 155; t += sampleRate)
{
Point p = Point(3 * t * t + 4 * (1 - t), 4 * t); //suppose the stroke function is like this
samplePoints.emplace_back(p);
}
}
bool Stroke::setStripIndx(int idx)
{
stripIndx = idx;
}
int Stroke::getStripIndx()
{
int stripIndx;
}
//VectorSketch.cpp
bool VectorSketch::Sample()
{
float sampleRate = 1.2 * strokeWidth;
for (auto stroke : Strokes)
{
stroke.getSamplePoints(sampleRate);
}
//remove hook artifacts
//....
}
bool IsCompatible(vector<Stroke>& a, vector<Stroke>& b)
{
return false;
}
//merge strip b to strip a
bool Merge(vector<Stroke>& a, vector<Stroke>& b)
{
int stripIdxOfa = a[0].getStripIndx();
for (auto bb : b)
{
//update the information of strokes of strip b, and update the elements of strip a
bb.setStripIndx(stripIdxOfa);
a.emplace_back(bb);
}
return true;
}
bool IsFormjectionByYin2022(vector<Stroke>& a, vector<Stroke>& b)
{
//2022 Yin Algorithms
//..
}
void FittingCurvesByVanMossel2021()
{
//2021 Van Mossel Algorithms
//...
}
bool VectorSketch::LocalConsolidation()
{
auto Classfier = std::make_unique<LocalClassifier>();
int stripIdx = -1;
for (int i = 0; i < Strokes.size(); i++)
{
Stroke curStroke = Strokes[i];
Stroke previousStroke = Strokes[i - 1];
vector<Stroke> previousStrip = Strips[previousStroke.getStripIndx()];
float probability = Classfier->getLikelihoodSP(curStroke, previousStrip);

//curStroke belongs to the sub-strip the previousStroke belongs to
if (probability > 0.5)
{
//update curstroke information
Strokes[i].setStripIndx(stripIdx);
//update the previous strip
previousStrip.emplace_back(curStroke);
//proceed to the next stroke
continue;
}
else
{
//save curStroke as a new sub-strip
Strokes[i].setStripIndx(++stripIdx);
Strips[stripIdx].emplace_back(curStroke);

//previousStrip has more than one stroke, decide if it can be merged with other strips.
if (previousStrip.size() > 1)
{
int p_max = 0;
int idx_max = -1;
for (int j = 0; j < Strips.size(); j++)
{

//only call the classifier if two sub-strips are deemed compatible
if (IsCompatible(Strips[j], previousStrip))
{
float p = Classfier->getLikelihoodPP(Strips[j], previousStrip);
if (p > p_max)
{
p_max = p;
idx_max = j;
}
}

}
//merge the most similar two strips to one strip
Merge(Strips[idx_max], previousStrip);
Strips[stripIdx].clear();
--stripIdx;
previousStrip.clear();

}
}
}
//after the iteration above, the whole strokes are grouped into different strips, as the differnt-color curves in the figure.

//fitting curves to these strips
FittingCurvesByVanMossel2021();
}
bool VectorSketch::GlobalConsolidation()
{
auto Classfier = std::make_unique<GlobalClassifier>();

//strip reevaluation
int n = Strips.size();
int StripIdx = n;
for (int i = 0; i < n; i++)
{
vector<Stroke> strip = Strips[i];
//find the least likelihood two seed strokes in this strip
int min_likelihood = INT_MAX;

vector<Stroke> seedStrokes(5); //Stroke seed_s1, seed_s2;
for (int k = 0; k < strip.size(); k++)
{
for (int l = k + 1; l < strip.size(); l++)
{
Stroke s1 = strip[k];
Stroke s2 = strip[l];
float p = Classfier->getLikelihoodSS(s1, s2);
if (p < min_likelihood)
{
min_likelihood = p;
seedStrokes[0] = s1;
seedStrokes[1] = s2;
}
}
}
//thie likelihood is high enougth, don't change this strip
if (min_likelihood > 0.6)
{
continue;
}
else
{
//Reassign other strokes to sub-strip formed by seed_s1 or sub-strip formed by seed_s2
int newStripIdx = StripIdx;

//seedStips..
vector<vector<Stroke>> seedStrips(5);
seedStrokes[1].setStripIndx(newStripIdx);
seedStrips[0].emplace_back(seedStrokes[0]); //seed_s1_strip
seedStrips[1].emplace_back(seedStrokes[1]); //seed_s2_strip

for (auto stroke : strip)
{
float p_max = 0;
for (auto sub_strip : seedStrips)
{
float pi = Classfier->getLikelihoodSP(stroke, sub_strip);
p_max = max(p_max, pi);
}
//this stroke doesn't belong to any seedStrip, so make this stroke as a new seed stroke
if (p_max < 0.5)
{
seedStrokes.emplace_back(stroke);
stroke.setStripIndx(++newStripIdx);
}
else
{
//assign this stroke to seed_s1_strip or seed_s2_strip or ..., according to the likelihood
//...
}
}
//merge this seedStrips according the likelihood

//update Strips, cause may produce new strips in step above
}
}

//Connectivity Preservation classifier in Yin et al.[2022]
for(auto strip : Strips)
for (auto stripb : Strips)
{
if (IsFormjectionByYin2022(strip, strip))
{
Merge(strip, strip);
}

}

//fitting curves to these strips
FittingCurvesByVanMossel2021();
}

The classifier can be implemented as follow:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
//RandomForestClassfier.cpp
float RandomForestClassfier::getLikelihoodPP(vector<Stroke>& a, vector<Stroke>& b)
{
//call the python module, the trained model..

/*
the classifier extract the features as follows:
//Angles and Distance
//Density
//Narrows and Side-by-Side Extent
//Evenness
//1d Parameterization Distortion
//Relative Precision
*/
return -1;
}
#

Comments

Your browser is out-of-date!

Update your browser to view this website correctly. Update my browser now

×