SVM regression (SVR) vs Linear Regression

1 min read

One particular nice thing about SVR is that the weight of each feature reflects the feature’s true contribution. If some features are highly correlated (this is common in neuroimaging data – e.g. adjacent voxel activity are very similar), you expect that the corresponding weights to be similar. SVR exactly does that. Linear regression, on the other hand, will give big weight to the feature which is best correlated to the dependent variable (y). Here is a demonstration in MatLab.

In the following simulation, we have 10 features which are highly correlated (see figure below).

Correlation between features
Correlation between features

And the variable to be predicted is fairly nicely correlated with the features. Below I plotted the weight of 10 features. The blue curve is for SVR, and the red is for linear regression. As you can see, the weight for SVR is smooth and balanced, but the weight for regression is like ‘winner takes all’, the 1st feature has a big weight but others very small.

Weight of Features (blue: SVR, red: regression)
Weight of Features (blue: SVR, red: regression)

Here are the source code in MatLab. normalize.m can be found at https://www.alivelearn.net/?p=1083

N = 1000;
M = 1;
t = randn(N,1);

clear r
m = 10;%1:10:100;
for M = m
x = [t];
for ii=1:M-1
    x = [x  t+ii*randn(N,1)/10];
end

x = normalize(x);

% t1 = randn(N,1);
% t2 = randn(N,1);
% x = [t1 t2];
% y = 3*t1 - 5*t2;
y = 2*t + randn(N,1)/2 + 0;

% corrcoef([x y]);
%
% b= glmfit(x,y);
for ii = 1
    for jj=1
        tic;model = svmtrain(y(1:N/2),x(1:N/2,:),['-s 4 -t 0 -n ' num2str(ii/2) ' -c ' num2str(1)]);toc
        tic;zz=svmpredict(y(N/2+1:end),x(N/2+1:end,:),model);toc
        tmp = corrcoef(zz, y(N/2+1:end));
        r(M) = tmp(2);
    end
end

w = model.SVs' * model.sv_coef;
b = -model.rho;
b1 = [w]

% regression
b2 = glmfit(x(1:N/2,:), y(1:N/2,:));
yy = x(N/2+1:end,:)*b2(2:end) + b2(1);
b2 = b2(2:end)

sum((zz - y(N/2+1:end)).^2)
sum((yy - y(N/2+1:end)).^2)

end

figure('color','w');imagesc(corrcoef(x));colorbar
caxis([0 1])
figure('color','w');plot(b1,'o-');hold on;plot(b2,'o-r');xlabel('feature index');ylabel('weight')

return

hold on;plot(m, r, 'ro-');xlabel('# of dimension'); ylabel('r')
figure('color','w');plot(m, r, 'ro-');

figure('color','w');plot(x(1:N/2,:), y(1:N/2), 'b.');
hold on;plot(x(N/2+1:end,:), zz, 'r.');
xlabel('x')
ylabel('y')
legend({'training','test'})

%figure('color','w'); plot(zz, y(N/2+1:end), '.'); axis equal;axis square;
%figure('color','w'); plot(zz - y(N/2+1:end), '.')



写作助手,把中式英语变成专业英文


Want to receive new post notification? 有新文章通知我

第五十期fNIRS Journal Club通知2024/03/30, 10am 王一晖

早期的 STEM 教育对于以后的学习至关重要。现有研究尚未就STEM教学法达成共识,学生先验知识对基于故事的STEM教学法的影响还有待探讨。来自澳门大学张娟教授团队的王一晖将会分享基于fNIRS超扫描
Wanling Zhu
9 sec read

第四十九期fNIRS Journal Club视频 高倩博士

Youtube: https://youtu.be/a2QlCFZUytA优酷: https://v.youku.com/v_show/id_XNjM3MjMyNjUxMg==.html 竞技比赛中,
Wanling Zhu
12 sec read

第四十九期fNIRS Journal Club通知2024/02/24, 10am 高倩博士

竞技比赛中,特别是需要注意高度集中的射击项目中,如何保持稳定的注意控制对运动员的表现至关重要。近期的一项研究调查了赛前短时正念练习是否对运动员在射击比赛中的注意控制具有提升作用。来自北京体育大学心理学
Wanling Zhu
9 sec read

2 Replies to “SVM regression (SVR) vs Linear Regression”

  1. Very interesting. I was running into a similar problem with regression and wonder what the underlying reason for the winner takes all phenomenon is. Do you know more about it?
    Thanks,
    Henning

  2. Great article!

    e-SVR seems also to work great on Binary Classification Problems like pedestrian detection (Better than SVC…). I am a bit confused since Regression is for continuous output whereas Classification is discrete-oriented. Could you give me some insights about that (SVR vs SVC) ?

    Cheers,

    Djébril

Leave a Reply to Henning Cancel reply

Your email address will not be published. Required fields are marked *