• J
    User Attivo •

    GEt Robots.txt 301 da bot google legittimo

    Ciao a tutti,

    dai log vedo che il file robots.txt viene richiamato da alcuni ip legittimi di google bots. Quelli che riscontro sono

    66.249.64.3
    66.249.64.5
    66.249.64.7
    66.249.64.72
    66.249.64.74

    Lo strano è che agli ultimi due il server risponde 301. Vi riporto un breve estratto. Perchè risponde 301 solo a quegli ips? Non ci sono regole particolari a quanto pare lato server. Succede anche a voi?

    [TABLE="width: 0"]

    [TD]66.249.64.5[/TD]
    [TD]-[/TD]
    [TD]-[/TD]
    [TD][05/Nov/2018:14:05:41[/TD]
    [TD]+0100][/TD]
    [TD]GET /robots.txt HTTP/1.0[/TD]
    [TD="align: right"] 200[/TD]
    [TD="align: right"]152[/TD]
    [TD]-[/TD]
    [TD]Mozilla/5.0 (compatible; Googlebot/2.1; [/TD]
    [/TR]

    [TD]66.249.64.5[/TD]
    [TD]-[/TD]
    [TD]-[/TD]
    [TD][05/Nov/2018:16:39:44[/TD]
    [TD]+0100][/TD]
    [TD]GET /robots.txt HTTP/1.0[/TD]
    [TD="align: right"] 200[/TD]
    [TD="align: right"]152[/TD]
    [TD]-[/TD]
    [TD]Mozilla/5.0 (compatible; Googlebot/2.1; [/TD]
    [/TR]

    [TD]66.249.64.7[/TD]
    [TD]-[/TD]
    [TD]-[/TD]
    [TD][06/Nov/2018:19:18:19[/TD]
    [TD]+0100][/TD]
    [TD]GET /robots.txt HTTP/1.0[/TD]
    [TD="align: right"] 200[/TD]
    [TD="align: right"]152[/TD]
    [TD]-[/TD]
    [TD]Mozilla/5.0 (compatible; Googlebot/2.1;[/TD]
    [/TR]

    [TD]66.249.64.72[/TD]
    [TD]-[/TD]
    [TD]-[/TD]
    [TD][02/Nov/2018:04:39:29[/TD]
    [TD]+0100][/TD]
    [TD]GET /robots.txt HTTP/1.0[/TD]
    [TD="align: right"]301[/TD]
    [TD="align: right"]395[/TD]
    [TD]-[/TD]
    [TD]Mozilla/5.0 (compatible; Googlebot/2.1; [/TD]
    [/TR]

    [TD]66.249.64.72[/TD]
    [TD]-[/TD]
    [TD]-[/TD]
    [TD][02/Nov/2018:11:03:33[/TD]
    [TD]+0100][/TD]
    [TD]GET /robots.txt HTTP/1.0[/TD]
    [TD="align: right"]301[/TD]
    [TD="align: right"]395[/TD]
    [TD]-[/TD]
    [TD]Mozilla/5.0 (compatible; Googlebot/2.1; [/TD]
    [/TR]

    [TD]66.249.64.72[/TD]
    [TD]-[/TD]
    [TD]-[/TD]
    [TD][04/Nov/2018:11:37:46[/TD]
    [TD]+0100][/TD]
    [TD]GET /robots.txt HTTP/1.0[/TD]
    [TD="align: right"]301[/TD]
    [TD="align: right"]395[/TD]
    [TD]-[/TD]
    [TD]Mozilla/5.0 (compatible; Googlebot/2.1; [/TD]
    [/TR]

    [TD]66.249.64.74[/TD]
    [TD]-[/TD]
    [TD]-[/TD]
    [TD][05/Nov/2018:05:25:04[/TD]
    [TD]+0100][/TD]
    [TD]GET /robots.txt HTTP/1.0[/TD]
    [TD="align: right"]301[/TD]
    [TD="align: right"]395[/TD]
    [TD]-[/TD]
    [TD]Mozilla/5.0 (compatible; Googlebot/2.1; [/TD]
    [/TR]

    [TD]66.249.64.74[/TD]
    [TD]-[/TD]
    [TD]-[/TD]
    [TD][06/Nov/2018:19:18:19[/TD]
    [TD]+0100][/TD]
    [TD]GET /robots.txt HTTP/1.0[/TD]
    [TD="align: right"]301[/TD]
    [TD="align: right"]395[/TD]
    [TD]-[/TD]
    [TD]Mozilla/5.0 (compatible; Googlebot/2.1; [/TD]
    [/TR]
    [/TABLE]

    0
  • giorgiotave
    Community Manager •

    Ciao John,
    la situazione è cambiata? Hai notato altro?

    0
Effettua l'accesso per rispondere