Читаем документацию:
Правильно будет вот так:To specify a file filter for this property, assign a string that consists of a filter description and a file mask separated with the vertical bar (|) character. Do not include spaces around the vertical bar.
form.OpenDialog1.Filter := 'Text files (*.txt)|*.txt';
Спасибо - помогло :)