You have the rules here with hits, so traffic was redirected. But there is no ipset in the matches so all traffic goes to the tor.
[74:3848] -A zone_lan_prerouting -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK SYN -m comment --comment "!fw3: Intercept-TCP" -j REDIRECT --to-ports 9040
[533:39775] -A zone_wan_postrouting -m comment --comment "!fw3: Custom wan postrouting rule chain" -j postrouting_wan_rule
So if no match in ipset then it should not reroute traffic. How it will be then? Some kind of if-else construction? Also torproject.org in ipset why it is not routed?
There is no ipset in the matching conditions of the rule, so everything matching (all tcp) is redirected.
Does the ipset populate anything?
ipset list routetotor
ipset list routetotor
Name: routetotor
Type: hash:ip
Revision: 4
Header: family inet hashsize 1024 maxelem 65536
Size in memory: 56
References: 0
Number of entries: 0
Members:
The iplist is empty. Go back here and verify you got all the steps right. If there are no entries in the iplist everything will be sent to tor. As for why some page is not working, this is something you'll have to troubleshoot in tor.
I don't think I missed any step. Here all steps I put together. Check it if any step missing :S
3. install packages for ipset
opkg update && opkg install kmod-ipt-ipset ipset && opkg remove dnsmasq && opkg install dnsmasq-full
4. Put prerouting in /etc/firewall.user
iptables -t nat -I prerouting_lan_rule -d 192.168.1.1 -j ACCEPT
5. Add ipset line in /etc/config/dhcp under dnsmasq
config dnsmasq
option domainneeded 1
option boguspriv 1
option filterwin2k 0 # enable for dial on demand
option localise_queries 1
option rebind_protection 1 # disable if upstream must serve RFC1918 addresses
option rebind_localhost 1 # enable for RBL checking and similar services
#list rebind_domain example.lan # whitelist RFC1918 responses for domains
option local '/lan/'
option domain 'lan'
option expandhosts 1
option nonegcache 0
option authoritative 1
option readethers 1
option leasefile '/tmp/dhcp.leases'
option resolvfile '/tmp/resolv.conf.auto'
#list server '/mycompany.local/1.2.3.4'
option nonwildcard 1 # bind to & keep track of interfaces
#list interface br-lan
#list notinterface lo
#list bogusnxdomain '64.94.110.11'
option localservice 1 # disable to allow DNS requests from non-local subnets
list ipset '/pastebin.com/torproject.org/routetotor'
6. Tor install and base setup
opkg update
opkg install tor
sed -i -e "
/^AutomapHostsOnResolve/s/^/#/
\$a AutomapHostsOnResolve 1
/^VirtualAddrNetwork/s/^/#/
\$a VirtualAddrNetworkIPv4 172.16.0.0/12
\$a VirtualAddrNetworkIPv6 fc00::/7
/^DNSPort/s/^/#/
\$a DNSPort 0.0.0.0:9053
\$a DNSPort [::]:9053
/^TransPort/s/^/#/
\$a TransPort 0.0.0.0:9040
\$a TransPort [::]:9040
" /etc/tor/torrc
/etc/init.d/tor restart
7. Firewall DNS and TCP intercepts setup
uci -q delete firewall.dns_int
uci set firewall.dns_int="redirect"
uci set firewall.dns_int.name="Intercept-DNS"
uci set firewall.dns_int.src="lan"
uci set firewall.dns_int.src_dport="53"
uci set firewall.dns_int.dest_port="9053"
uci set firewall.dns_int.family="ipv4"
uci set firewall.dns_int.proto="udp"
uci set firewall.dns_int.target="DNAT"
uci -q delete firewall.tcp_int
uci set firewall.tcp_int="redirect"
uci set firewall.tcp_int.name="Intercept-TCP"
uci set firewall.tcp_int.src="lan"
uci set firewall.tcp_int.dest_port="9040"
uci set firewall.tcp_int.family="ipv4"
uci set firewall.tcp_int.proto="tcp"
uci set firewall.tcp_int.extra="--syn"
uci set firewall.tcp_int.target="DNAT"
uci commit firewall
/etc/init.d/firewall restart
8. Adding ipset route to /etc/config/firewall file
config ipset
option enabled '1'
option name 'routetotor'
option match 'ip'
option storage 'hash'
config redirect
option src 'LAN'
option name 'TorHTTP'
option dest 'wan'
option target 'DNAT'
option ipset 'routetotor dest'
list proto 'tcp'
list proto 'udp'
option src_dport '80'
option dest_port '9040'
config redirect
option src 'LAN'
option name 'TorHTTPs'
option dest 'wan'
option target 'DNAT'
option ipset 'routetotor dest'
list proto 'tcp'
list proto 'udp'
option src_dport '443'
option dest_port '9040'
9. Reboot router and done.
reboot
Lower case letters, linux is case sensitive. I fixed the wiki to avoid confusion. It would help though to restart the firewall instead of rebooting, you would see the error message.
I installed dnsmasq-full and followed the steps in the page and I have the line in the firewall
[0:0] -A zone_lan_prerouting -p tcp -m tcp --dport 80 -m set --match-set routetotor dst -m comment --comment "!fw3: TorHTTP" -j REDIRECT --to-ports 9040
I lowered case and reboot router. I see these lines in ipset output:
[0:0] -A zone_lan_prerouting -p tcp -m tcp --dport 80 -m set --match-set routetotor dst -m comment --comment "!fw3: TorHTTP" -j REDIRECT --to-ports 9040
[0:0] -A zone_lan_prerouting -p udp -m udp --dport 80 -m set --match-set routetotor dst -m comment --comment "!fw3: TorHTTP" -j REDIRECT --to-ports 9040
[0:0] -A zone_lan_prerouting -p tcp -m tcp --dport 443 -m set --match-set routetotor dst -m comment --comment "!fw3: TorHTTPs" -j REDIRECT --to-ports 9040
[0:0] -A zone_lan_prerouting -p udp -m udp --dport 443 -m set --match-set routetotor dst -m comment --comment "!fw3: TorHTTPs" -j REDIRECT --to-ports 9040
Still youtube send me .de and ask chapta :S
Are there any hits in the iptables rules? I don't see any, all are [0:0]
Which IPs are in the ipset? ipset list routetotor
Which IPs are resolved for youtube? host youtube.com
ipset list routetotor:
Name: routetotor
Type: hash:ip
Revision: 4
Header: family inet hashsize 1024 maxelem 65536
Size in memory: 136
References: 4
Number of entries: 2
Members:
104.23.99.190
104.23.98.190
-ash: host: not found
Use some other command to resolve the name if you don't have host, like dig or nslookup.
nslookup youtube.com
Server: 127.0.0.1
Address: 127.0.0.1#53
Name: youtube.com
Address 1: 172.217.23.46
Address 2: 2a00:1450:4016:804::200e
it give output to return internal ip
So youtube is not sent to tor. What seems to be the problem?
Today I am in Russia. Yesterday I was in Germany. Where will I be tomorrow I wonder. 
Also this:
Oh I am again in Germany:
can we tie TCP redirect to ipset too? When I remove below from firewall pages works like no tor at all.
config redirect 'tcp_int'
option name 'Intercept-TCP'
option src 'lan'
option dest_port '9040'
option family 'ipv4'
option proto 'tcp'
option extra '--syn'
option target 'DNAT'
I cannot help you much more unfortunately. You should have already understood how is the redirect and the ipset working.
Verify in tor server that only pages meant for tor are redirected.
Compare the IP addresses in the IPset with the IP addresses that you resolve for the sites you want to send to tor and sites that shouldn't go to tor. If some IP is used by both, then both sites will be sent to tor.
This is sending all the tcp traffic from lan to tor unconditionally. I don't remember advising you to use it, so I am not sure what is it doing there. If you want to access more protocols from tor you can use it.
YESS!! THAT WORK THANKS TRENDY!!! YOU ARE SO TRENDY 
config redirect 'dns_int'
option name 'Intercept-DNS'
option src 'lan'
option src_dport '53'
option dest_port '9053'
option family 'ipv4'
option proto 'udp'
option target 'DNAT'
option ipset 'routetotor dest'
config redirect 'tcp_int'
option name 'Intercept-TCP'
option src 'lan'
option dest_port '9040'
option family 'ipv4'
option proto 'tcp'
option extra '--syn'
option target 'DNAT'
option ipset 'routetotor dest'
Tied DNS and TCP intercept to ipset. Now ONLY selected address go to tor route and ask chapta which is normal when use tor :). Other sites like youtube have direct connection to web. Should I add this to TOR client page? or start new page?
Do you have any hits on the first rule? I find it hard that it is actually working.
iptables-save -c | grep 9053
this is output but I just reboot router. Did we set logging on?
iptables-save -c | grep 9053
[0:0] -A zone_lan_prerouting -p udp -m udp --dport 53 -m set --match-set routetotor dst -m comment --comment "!fw3: Intercept-DNS" -j REDIRECT --to-ports 9053
and https port:
iptables-save -c | grep 9040
[1:52] -A zone_lan_prerouting -p tcp -m set --match-set routetotor dst -m tcp --tcp-flags FIN,SYN,RST,ACK SYN -m comment --comment "!fw3: Intercept-TCP" -j REDIRECT --to-ports 9040
[0:0] -A zone_lan_prerouting -p tcp -m tcp --dport 80 -m set --match-set routetotor dst -m comment --comment "!fw3: TorHTTP" -j REDIRECT --to-ports 9040
[0:0] -A zone_lan_prerouting -p udp -m udp --dport 80 -m set --match-set routetotor dst -m comment --comment "!fw3: TorHTTP" -j REDIRECT --to-ports 9040
[0:0] -A zone_lan_prerouting -p tcp -m tcp --dport 443 -m set --match-set routetotor dst -m comment --comment "!fw3: TorHTTPs" -j REDIRECT --to-ports 9040
[0:0] -A zone_lan_prerouting -p udp -m udp --dport 443 -m set --match-set routetotor dst -m comment --comment "!fw3: TorHTTPs" -j REDIRECT --to-ports 9040
and still check.torproject.org unreachable. I think it is tor problem at this moment. Can't reroute internal request I guess?
Again:
iptables-save -c | grep 9040
[2:104] -A zone_lan_prerouting -p tcp -m set --match-set routetotor dst -m tcp --tcp-flags FIN,SYN,RST,ACK SYN -m comment --comment "!fw3: Intercept-TCP" -j REDIRECT --to-ports 9040
[0:0] -A zone_lan_prerouting -p tcp -m tcp --dport 80 -m set --match-set routetotor dst -m comment --comment "!fw3: TorHTTP" -j REDIRECT --to-ports 9040
[0:0] -A zone_lan_prerouting -p udp -m udp --dport 80 -m set --match-set routetotor dst -m comment --comment "!fw3: TorHTTP" -j REDIRECT --to-ports 9040
[0:0] -A zone_lan_prerouting -p tcp -m tcp --dport 443 -m set --match-set routetotor dst -m comment --comment "!fw3: TorHTTPs" -j REDIRECT --to-ports 9040
[0:0] -A zone_lan_prerouting -p udp -m udp --dport 443 -m set --match-set routetotor dst -m comment --comment "!fw3: TorHTTPs" -j REDIRECT --to-ports 9040